-- Logs begin at Sun 2022-07-31 00:10:28 UTC, end at Sun 2022-07-31 01:33:01 UTC. -- Jul 31 00:53:05.049707 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Jul 31 00:53:05.057308 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Jul 31 00:53:05.059319 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Jul 31 00:53:05 2022] *** Jul 31 00:53:05.059319 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Jul 31 00:53:05.059319 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Jul 31 00:53:05.059319 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: nodename: dv-cinder-master-3-iscsi Jul 31 00:53:05.059319 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: machine: x86_64 Jul 31 00:53:05.059319 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: clock source: unix Jul 31 00:53:05.059319 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: pcre jit disabled Jul 31 00:53:05.059319 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: detected number of CPU cores: 4 Jul 31 00:53:05.059815 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: current working directory: / Jul 31 00:53:05.059815 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: detected binary path: /usr/bin/uwsgi-core Jul 31 00:53:05.059815 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: your processes number limit is 47680 Jul 31 00:53:05.061516 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: your memory page size is 4096 bytes Jul 31 00:53:05.061516 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: detected max file descriptor number: 2048 Jul 31 00:53:05.061516 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: lock engine: pthread robust mutexes Jul 31 00:53:05.061516 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: thunder lock: enabled Jul 31 00:53:05.061516 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Jul 31 00:53:05.061516 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Jul 31 00:53:05.101017 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: Python main interpreter initialized at 0x563d73a909c0 Jul 31 00:53:05.101017 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: python threads support enabled Jul 31 00:53:05.101017 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: your server socket listen backlog is limited to 100 connections Jul 31 00:53:05.101017 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: your mercy for graceful operations on workers is 90 seconds Jul 31 00:53:05.101415 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: mapped 403077 bytes (393 KB) for 2 cores Jul 31 00:53:05.101485 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: *** Operational MODE: preforking *** Jul 31 00:53:05.101719 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: *** uWSGI is running in multiple interpreter mode *** Jul 31 00:53:05.101719 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: spawned uWSGI master process (pid: 80203) Jul 31 00:53:05.101923 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Jul 31 00:53:05.103014 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: spawned uWSGI worker 1 (pid: 80208, cores: 1) Jul 31 00:53:05.103014 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: spawned uWSGI worker 2 (pid: 80209, cores: 1) Jul 31 00:53:05.103014 dv-cinder-master-3-iscsi devstack@keystone.service[80203]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Jul 31 00:53:10.619051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 31 00:53:10.619051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: from cryptography.utils import int_from_bytes Jul 31 00:53:10.619872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 31 00:53:10.619872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: from cryptography.utils import int_from_bytes Jul 31 00:53:10.625710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 31 00:53:10.625710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: from cryptography.utils import int_from_bytes Jul 31 00:53:10.626191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 31 00:53:10.626191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: from cryptography.utils import int_from_bytes Jul 31 00:53:10.669605 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 31 00:53:10.669808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 31 00:53:10.669998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] command line args: [] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 31 00:53:10.670273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 31 00:53:10.670449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ================================================================================ {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 31 00:53:10.670709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] admin_token = **** {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.670955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] config_dir = [] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.671140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.671321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] config_source = [] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.671526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] debug = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.671723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.673114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.673114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.673114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.673114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.673114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] list_limit = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.673114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] log_config_append = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.673522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.673522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] log_dir = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.673652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 31 00:53:10.673724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] log_file = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.673792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 31 00:53:10.673862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.673941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] command line args: [] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 31 00:53:10.674054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.674174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 31 00:53:10.674273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ================================================================================ {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 31 00:53:10.674395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.674529 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.674615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] admin_token = **** {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.674706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.674790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] config_dir = [] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.674907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.674988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.675131 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.675214 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] config_source = [] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.675329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] debug = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.675463 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.675549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.675654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.675761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.676098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.676198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.676325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.676450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.676536 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.676698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.676782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.676928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] list_limit = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.677014 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.677093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] log_config_append = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.677251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.677340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.677418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] log_dir = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.677595 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] log_file = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.677751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.677841 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.677935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] publish_errors = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.678066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.678168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.678314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.678397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.678561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.678654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.678727 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.678808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.678908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.679023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.679153 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.679255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] standard_threads = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.679420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.679514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.679657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.679736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.679889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.679970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.680047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] use_journal = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.680116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.680220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] use_json = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.680340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.680424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] use_stderr = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.680552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] use_syslog = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.680629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.680780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.680864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.681032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.681110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.681524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.681603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.681731 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] publish_errors = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.681808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.681964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.682042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.682194 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.682277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.682373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.682450 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.682618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.682701 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.682851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.682928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.683085 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] standard_threads = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.683162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.683315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.683418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.683585 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.683671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.683763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.683840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.683940 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] use_journal = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.684061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.684139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] use_json = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.684276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] use_stderr = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.684352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.684451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] use_syslog = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.684538 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 31 00:53:10.684766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.684936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.685013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.685177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.685262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.685436 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.685514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.685677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.685755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.685923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.686009 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.686158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.686235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.686404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.686487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.686640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.686795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.686870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.686939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.687084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.687161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.687312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.687440 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.687601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.687848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.687989 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.688069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.688243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.688329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.688495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.688573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.688728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.688806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.688961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.689047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.689209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.689287 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.689455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.689533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.689691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.689780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.689938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.690015 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.690167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.690243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.690401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.690485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.690634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.691198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.691402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.691545 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.691660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.691773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.691889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.692010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.692119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.692248 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.692378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.692499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.692578 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.692750 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.692826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.692981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.693135 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.693215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.693292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.693446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.693530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.693688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.693773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.693923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.694000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.694151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.694233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.694386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.694472 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.694618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.694700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.694849 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.694925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.695080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.695165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.696203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.696371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.696371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.696581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.696696 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.696822 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.696936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.697050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.697352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.697478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.697613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.697724 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.697834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.697938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.698061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.698166 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.698300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.698416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.698542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.698620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.698779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.698898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.699025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.699114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.699270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.699371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.699536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.699624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.699777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.699895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.700012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.700091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.700248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.700331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.700478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.700555 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.700723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.700805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.700959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.701046 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.701191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.701269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.701423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.701502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.701656 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.701744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.701892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.701969 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.702124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.702204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.702359 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.702442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.702592 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.702711 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.702850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.702930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.703087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.703164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.703323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.703420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.703588 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.703669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.703824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.703902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.704061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.704138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.704294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.704374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.704527 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.704606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.704757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.704835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.704986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.705065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.705213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.705291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.705444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.705522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.705716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.705796 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.705896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.705974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.706128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.706206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.706406 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.707036 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.707203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.707290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.707470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.707565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.707714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.707793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.707953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.708033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.708199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.708278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] auth.external = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.708427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.708504 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.708662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.708741 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.708897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.708982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.709128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.709216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] auth.password = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.709360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.709438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] auth.token = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.709590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.709668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.709821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.709909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.710047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.710124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.710283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.710359 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.710515 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.710628 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.710759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.710847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.710989 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.711074 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.711222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] credential.caching = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.711301 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.711489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.711583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.711738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.711818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.711967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] auth.external = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.712045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.712199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.712289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.712438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.712517 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.712671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.712750 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.712932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.713019 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] auth.password = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.713161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] auth.token = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.713239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.713401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.713480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.713639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.713726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.713888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.713967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.714119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.714197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.714367 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.714457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.714617 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.714697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.714848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.714934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.715084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] credential.caching = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.715171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] federation.caching = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.715331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.715485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.715597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.715677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.715843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.715935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.716074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.716155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.716314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.716392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.716546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.716632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.716797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.716876 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.717031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.717111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.717279 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.717365 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.717511 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.717589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.717765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.717843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.717998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.caching = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.718085 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.718248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.718339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.718486 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.718566 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.718721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.718809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.718954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.719033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] federation.caching = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.719200 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.719279 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.719453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.719548 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.719690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.719769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.719935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.720014 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.720182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.720334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.720409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.720480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.720624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.720710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.720850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.720931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.721091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.721170 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.721334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.721413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.721578 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.721658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.721812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.caching = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.721889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.722082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.722159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.722330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.722409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.722570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.722650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.722802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.722879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.723030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.723110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.723263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.723342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.723513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.723602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.723745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.723826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.723981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.724061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.724218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.724297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.724451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.724532 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.724678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.724755 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.724921 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.724998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.725162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.725241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.725395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.725473 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.725639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.725718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.725883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.725964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.726111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.726191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.726343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.726422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.726570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.726650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.726804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.726883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.727038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.727116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.727266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.727370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.727525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.727611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.727758 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.727883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.727996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.728076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.728238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.728315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.728468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.728546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.728692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.728771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.728937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.729017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.729169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.729247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.729402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.729481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.729666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.729742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.729864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.729934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.730101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.730180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.730327 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.730405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.730567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.730651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.730794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.730873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.731217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.731312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.731406 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.731481 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.731566 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.731641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.731754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.731834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.731980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.732068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.732211 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.732291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.732440 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.732518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.732670 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.732756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.732896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.732976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.733140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.733225 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.733374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.733460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.733610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.733690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.733842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.733976 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.734080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.734231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.734339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.734461 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.734571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.734707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.734818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.734942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.735065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.735176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.735311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.735458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.735570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.735662 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.735800 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.735879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.736034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.736113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.736268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.736356 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.736503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.736582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.736733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.736812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.736985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.737073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.737232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.737311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.737461 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.737538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.738229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.738319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.738474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.738553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.738701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.738778 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.738928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.739014 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.739162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.739241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.739414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.739495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] resource.caching = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.739663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.739753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.739899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.739977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.740131 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.740209 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.740378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.740466 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.740612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.740697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.740847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.740925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.741081 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.741166 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.741312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.741390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.741547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.741626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.741778 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] role.caching = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.741869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.742012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.742090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] role.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.742236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.742381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.742480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.742629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.742709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.742864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.742943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.743104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.743183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.743338 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] resource.caching = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.743473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.743609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.743699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.743859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.743938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.744096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.744177 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.744346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.744424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.744595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.744675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.744829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.744908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.745072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.745150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.745309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.745395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.745549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.745730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.745809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] role.caching = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.745964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.746043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] role.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.746211 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.746298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.746458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.746538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.746702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.746781 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.746946 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.747025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.747175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.747253 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.747455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.747536 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.747705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.747786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.747940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.748018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.748169 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.748248 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.748396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.748477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.748624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.748703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.748871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.748947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.749119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.749198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.749361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.749438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.749595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.749778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.749865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.750016 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.750095 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.750259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] token.caching = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.750335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.750491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.750581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.750735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.750805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.750968 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.751047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.751201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.751280 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.751475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.751557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.751709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.751787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.751950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.752031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.752180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.752258 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.752412 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.752497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.752644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.752723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.752878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.752958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.753109 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.753189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.753902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.753991 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.754145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.754214 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.754370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.754474 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.754641 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.754721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.754879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] token.caching = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.754959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.755119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.755198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.connection = **** {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.755382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.755479 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.755624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.755706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.755882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.755962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.756119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.756197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.756354 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.756435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.756588 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.756669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.756822 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.756948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.757057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.757139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.757299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.757377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.757529 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.757607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.757762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.757841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.757998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.758115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.758241 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.758321 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.758469 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.758543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.758707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.758785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.758933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.759012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.759164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.759244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.connection = **** {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.759406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.759522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.759604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80208) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 31 00:53:10.759772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.760011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.760252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.760487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.760724 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.760960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.761198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.761442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.761678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.761921 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.762155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.762401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.762634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.762877 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.763106 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.763345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.763602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 31 00:53:10.763773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80209) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 31 00:53:10.788448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.788675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.788900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.791864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.792097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.792325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.797181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.801471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.828273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.828500 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.831094 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.831379 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.882482 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.882673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.933359 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.933586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.934067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:10.934305 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.004679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.004906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.005128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.005349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.005497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.005585 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.005727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.005811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.005953 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.006169 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.006385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.006600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.029422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.029672 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.029774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.029944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.030031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.030178 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.030262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.030400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.030493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.030616 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.030698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.030829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.030912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.031038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.042114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.042501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.055888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.056015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.056083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.056254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.056340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.056480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.056563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.056695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.056793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.056910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.056992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.057122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80209) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.057203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.057330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80208) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 31 00:53:11.074788 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.074997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 31 00:53:11.075133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.075335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.075440 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 31 00:53:11.075561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 31 00:53:11.075697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.075832 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.075914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 31 00:53:11.076093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.076184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.076350 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.076432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.076611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.076691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.076864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.086341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.086441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.086620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.086713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.086887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.086979 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.087167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.087250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.093983 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.094076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.099247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.099383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.099547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.099630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.099827 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.099909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.100104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.100225 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.100380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.100463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.100634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.100729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.100895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.100977 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.101160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.101240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.101420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.101509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.101686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.101766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.101944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.102024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.125156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.125279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.126638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.126724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.126924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.127025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.131518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.131607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.131793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.131876 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.132067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.132163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.138536 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.138631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.138815 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.138898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.142931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.143224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.143433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.143528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.143714 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.143810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.144026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.144109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.144304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.144387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.144575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.144659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.144852 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.144933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.145118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.145396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.161102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.161381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.161583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.161861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.165299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.165725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.168435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.168741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.168821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.169099 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.173079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.173419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.177128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.177357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.180284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.180481 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.180565 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.180774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.180856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.181054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.181145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.181347 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.181430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.181629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.181711 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.181908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.197427 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.197612 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.197695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.197891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.197973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.198154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.198236 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.198424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.205005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.205322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.206560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.206832 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.206948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.207112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.209646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.209897 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.209993 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.210170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.210252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.210461 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.210544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.210738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.210822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.211023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.222562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.222754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.222866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.223071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.223171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.223304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.223424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.223603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.223688 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.223874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.223997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.224165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.224273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.224413 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.241453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.244623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.247293 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.247534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.250083 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.250250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.251589 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.251879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.252183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.253190 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.255829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.257366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.257656 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.257935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.258746 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.260333 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.263199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.263510 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.263790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.264064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.264352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.265933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.268859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.269152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.269432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.269712 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.272087 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.272370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.272645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.277667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.277955 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.278234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.279340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80208) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.279636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.279912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.280194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.280467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.280743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.281008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.281278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.281659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.281942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.282221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.282482 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.282757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.283014 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80208) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.285552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80209) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 31 00:53:11.285826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.286103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.286376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.286655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.286922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.287192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.287479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.287859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.288143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.288432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.288702 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.288964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.289220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80209) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 31 00:53:11.317090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x563d73a909c0 pid: 80208 (default app) Jul 31 00:53:11.320437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40eef30d-00ea-4ac3-849b-2d966a61bfb6 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:11.320719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40eef30d-00ea-4ac3-849b-2d966a61bfb6 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:11.320992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40eef30d-00ea-4ac3-849b-2d966a61bfb6 None None] PATH_INFO: `/v3` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:11.322771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x563d73a909c0 pid: 80209 (default app) Jul 31 00:53:11.325337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1/1] 10.222.0.42 () {50 vars in 791 bytes} [Sun Jul 31 00:53:11 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:19.077922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e289184-de53-4e30-9fbd-e242f1d37e10 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:19.078504 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e289184-de53-4e30-9fbd-e242f1d37e10 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:19.078504 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e289184-de53-4e30-9fbd-e242f1d37e10 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:19.083158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1/2] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:19 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:19.088873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d191cb54-f050-4eba-9dc6-c54ee8f66256 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:19.089356 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d191cb54-f050-4eba-9dc6-c54ee8f66256 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:19.089446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d191cb54-f050-4eba-9dc6-c54ee8f66256 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:19.158814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: INFO dbcounter [None req-d191cb54-f050-4eba-9dc6-c54ee8f66256 None None] Registered counter for database keystone Jul 31 00:53:19.178931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG oslo_db.sqlalchemy.engines [None req-d191cb54-f050-4eba-9dc6-c54ee8f66256 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80208) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 31 00:53:19.185662 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writer thread running {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 31 00:53:19.370283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG passlib.handlers.bcrypt [None req-d191cb54-f050-4eba-9dc6-c54ee8f66256 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80208) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 31 00:53:19.370595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG passlib.handlers.bcrypt [None req-d191cb54-f050-4eba-9dc6-c54ee8f66256 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80208) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 31 00:53:19.426913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d191cb54-f050-4eba-9dc6-c54ee8f66256 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:19.428542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d191cb54-f050-4eba-9dc6-c54ee8f66256 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:19.562218 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 2/3] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:19 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 474 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 31 00:53:19.713805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e289184-de53-4e30-9fbd-e242f1d37e10 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:19.782375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: INFO dbcounter [None req-0e289184-de53-4e30-9fbd-e242f1d37e10 None None] Registered counter for database keystone Jul 31 00:53:19.804455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG oslo_db.sqlalchemy.engines [None req-0e289184-de53-4e30-9fbd-e242f1d37e10 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80209) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 31 00:53:19.810607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writer thread running {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 31 00:53:19.937477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e289184-de53-4e30-9fbd-e242f1d37e10 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:19.941771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b3869a-327e-4cd5-b018-a71cc3ed1cf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:19.942400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b3869a-327e-4cd5-b018-a71cc3ed1cf0 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:19.942684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b3869a-327e-4cd5-b018-a71cc3ed1cf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:19.942961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b3869a-327e-4cd5-b018-a71cc3ed1cf0 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:19.951486 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b3869a-327e-4cd5-b018-a71cc3ed1cf0 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:20.003992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: WARNING keystone.common.rbac_enforcer.enforcer [None req-e2b3869a-327e-4cd5-b018-a71cc3ed1cf0 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 31 00:53:20.004252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b3869a-327e-4cd5-b018-a71cc3ed1cf0 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:20.014303 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 2/4] 10.222.0.42 () {58 vars in 1324 bytes} [Sun Jul 31 00:53:19 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 31 00:53:20.021093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d191cb54-f050-4eba-9dc6-c54ee8f66256 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:20.035971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d191cb54-f050-4eba-9dc6-c54ee8f66256 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:20.040284 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17b2e74-a60a-44d2-9417-08052db0cce8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:20.041392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17b2e74-a60a-44d2-9417-08052db0cce8 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:20.041675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17b2e74-a60a-44d2-9417-08052db0cce8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:20.041949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17b2e74-a60a-44d2-9417-08052db0cce8 admin admin] PATH_INFO: `/v3/projects/18e627c596bb4c99b66fda211d0e4785` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:20.043687 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17b2e74-a60a-44d2-9417-08052db0cce8 admin admin] RBAC: Authorizing `identity:get_project(project_id=18e627c596bb4c99b66fda211d0e4785)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:20.096163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: WARNING keystone.common.rbac_enforcer.enforcer [None req-b17b2e74-a60a-44d2-9417-08052db0cce8 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 31 00:53:20.096785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17b2e74-a60a-44d2-9417-08052db0cce8 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:20.101326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 3/5] 10.222.0.42 () {56 vars in 1207 bytes} [Sun Jul 31 00:53:20 2022] GET /identity/v3/projects/18e627c596bb4c99b66fda211d0e4785 => generated 344 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:22.105602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dfbf69-ab5a-48b3-a182-27f764c532e9 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:22.106139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dfbf69-ab5a-48b3-a182-27f764c532e9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:22.106139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dfbf69-ab5a-48b3-a182-27f764c532e9 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:22.110277 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 3/6] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:22 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:53:22.115702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381c0ce9-421b-4262-9ac2-ec8959fc65d5 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:22.116089 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381c0ce9-421b-4262-9ac2-ec8959fc65d5 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:22.116260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381c0ce9-421b-4262-9ac2-ec8959fc65d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:22.135863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-381c0ce9-421b-4262-9ac2-ec8959fc65d5 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:22.137362 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-381c0ce9-421b-4262-9ac2-ec8959fc65d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:22.145705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 4/7] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:22 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 31 00:53:22.167809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dfbf69-ab5a-48b3-a182-27f764c532e9 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:22.180187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dfbf69-ab5a-48b3-a182-27f764c532e9 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:22.184402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90d3fb7-8429-4429-b7bd-9361b53f215b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:22.185009 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90d3fb7-8429-4429-b7bd-9361b53f215b admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:22.185291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90d3fb7-8429-4429-b7bd-9361b53f215b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:22.185564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90d3fb7-8429-4429-b7bd-9361b53f215b admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:22.193353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90d3fb7-8429-4429-b7bd-9361b53f215b admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:22.195135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90d3fb7-8429-4429-b7bd-9361b53f215b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:22.203972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 4/8] 10.222.0.42 () {58 vars in 1324 bytes} [Sun Jul 31 00:53:22 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 00:53:22.210249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381c0ce9-421b-4262-9ac2-ec8959fc65d5 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:22.219398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381c0ce9-421b-4262-9ac2-ec8959fc65d5 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:22.224385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b48b56-f648-490d-8603-206bb1f7f973 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:22.225012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b48b56-f648-490d-8603-206bb1f7f973 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:22.225293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b48b56-f648-490d-8603-206bb1f7f973 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:22.225568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b48b56-f648-490d-8603-206bb1f7f973 admin admin] PATH_INFO: `/v3/users/e711c7fc76434a0b86b92dc8bcf69e87` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:22.226915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b48b56-f648-490d-8603-206bb1f7f973 admin admin] RBAC: Authorizing `identity:get_user(user_id=e711c7fc76434a0b86b92dc8bcf69e87)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:22.228786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b48b56-f648-490d-8603-206bb1f7f973 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:22.230352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 5/9] 10.222.0.42 () {56 vars in 1198 bytes} [Sun Jul 31 00:53:22 2022] GET /identity/v3/users/e711c7fc76434a0b86b92dc8bcf69e87 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:24.306882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eccc55b-e8c4-40a4-b519-8d2cbbd2dcaa None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:24.307489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eccc55b-e8c4-40a4-b519-8d2cbbd2dcaa None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:24.307489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eccc55b-e8c4-40a4-b519-8d2cbbd2dcaa None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:24.308808 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 5/10] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:24.314150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11ece71-e5c2-44ea-8355-ada53b6a265e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:24.315521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11ece71-e5c2-44ea-8355-ada53b6a265e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:24.315521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11ece71-e5c2-44ea-8355-ada53b6a265e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:24.335635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c11ece71-e5c2-44ea-8355-ada53b6a265e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:24.336833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c11ece71-e5c2-44ea-8355-ada53b6a265e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:24.354655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05cd032-8999-46d2-b9aa-2613db3ca561 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:24.354944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05cd032-8999-46d2-b9aa-2613db3ca561 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:24.355242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05cd032-8999-46d2-b9aa-2613db3ca561 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:24.356607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 6/11] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:53:24.361759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93492ca1-2833-4fe3-9621-5369fafba979 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:24.362120 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93492ca1-2833-4fe3-9621-5369fafba979 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:24.362998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93492ca1-2833-4fe3-9621-5369fafba979 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:24.407853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG passlib.handlers.bcrypt [None req-93492ca1-2833-4fe3-9621-5369fafba979 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80209) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 31 00:53:24.408279 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG passlib.handlers.bcrypt [None req-93492ca1-2833-4fe3-9621-5369fafba979 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80209) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 31 00:53:24.430177 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 6/12] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:24.453523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11ece71-e5c2-44ea-8355-ada53b6a265e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:24.454635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-93492ca1-2833-4fe3-9621-5369fafba979 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:24.456350 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-93492ca1-2833-4fe3-9621-5369fafba979 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:24.480366 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:24.566345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 7/13] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:53:24.574677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11ece71-e5c2-44ea-8355-ada53b6a265e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:24.588541 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93492ca1-2833-4fe3-9621-5369fafba979 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:24.610486 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a854d040-8dc2-4231-840d-887061aea3f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:24.611130 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a854d040-8dc2-4231-840d-887061aea3f2 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:24.611437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a854d040-8dc2-4231-840d-887061aea3f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:24.611715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a854d040-8dc2-4231-840d-887061aea3f2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:24.624081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a854d040-8dc2-4231-840d-887061aea3f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-a854d040-8dc2-4231-840d-887061aea3f2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:24.633074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:24.634004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:53:24.634795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 7/14] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:53:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:24.640190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a854d040-8dc2-4231-840d-887061aea3f2 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:24.703986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93492ca1-2833-4fe3-9621-5369fafba979 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:24.708986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:24.733548 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:24.752851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a854d040-8dc2-4231-840d-887061aea3f2 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:24.753668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b637dfd-603f-4081-81af-08736f635605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:24.754434 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b637dfd-603f-4081-81af-08736f635605 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:24.754754 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b637dfd-603f-4081-81af-08736f635605 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:24.755031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b637dfd-603f-4081-81af-08736f635605 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:24.790120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d39b16-7dcc-4b09-85f7-c144886a1727 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:24.790731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d39b16-7dcc-4b09-85f7-c144886a1727 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:24.791006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d39b16-7dcc-4b09-85f7-c144886a1727 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:24.791277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d39b16-7dcc-4b09-85f7-c144886a1727 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:24.792019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d39b16-7dcc-4b09-85f7-c144886a1727 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:24.792320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d39b16-7dcc-4b09-85f7-c144886a1727 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:24.793725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d39b16-7dcc-4b09-85f7-c144886a1727 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:24.809304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 8/15] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:24.814925 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d39b16-7dcc-4b09-85f7-c144886a1727 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:24.831896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b637dfd-603f-4081-81af-08736f635605 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:24.833356 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b637dfd-603f-4081-81af-08736f635605 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:24.954599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 8/16] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 31 00:53:24.961321 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b637dfd-603f-4081-81af-08736f635605 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:24.965096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:25.052290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d39b16-7dcc-4b09-85f7-c144886a1727 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:25.082702 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b637dfd-603f-4081-81af-08736f635605 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:25.086933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb2b010-d847-477f-8092-3b6412b8476d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:25.087619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb2b010-d847-477f-8092-3b6412b8476d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:25.087901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb2b010-d847-477f-8092-3b6412b8476d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:25.088179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb2b010-d847-477f-8092-3b6412b8476d None admin] PATH_INFO: `/v3/users/e711c7fc76434a0b86b92dc8bcf69e87` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:25.089343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb2b010-d847-477f-8092-3b6412b8476d None admin] RBAC: Authorizing `identity:get_user(user_id=e711c7fc76434a0b86b92dc8bcf69e87)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:25.090812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb2b010-d847-477f-8092-3b6412b8476d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:25.092453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 9/17] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:24 2022] GET /identity/v3/users/e711c7fc76434a0b86b92dc8bcf69e87 => generated 247 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:25.098457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb2b010-d847-477f-8092-3b6412b8476d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:25.113442 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:25.119984 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57034a6-220c-42ad-95ca-2d4af7ccf583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:25.120882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57034a6-220c-42ad-95ca-2d4af7ccf583 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:25.121160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57034a6-220c-42ad-95ca-2d4af7ccf583 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:25.121430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57034a6-220c-42ad-95ca-2d4af7ccf583 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:25.140027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57034a6-220c-42ad-95ca-2d4af7ccf583 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:25.141513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57034a6-220c-42ad-95ca-2d4af7ccf583 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-f57034a6-220c-42ad-95ca-2d4af7ccf583 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:53:25.160880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:25.161918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:25.162772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:53:25.162772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:53:25.162772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:53:25.162772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:53:25.162772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 9/18] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:53:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:25.167918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57034a6-220c-42ad-95ca-2d4af7ccf583 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:25.202791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb2b010-d847-477f-8092-3b6412b8476d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:25.226599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:25.238327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db814e4-5802-4764-b121-2451c13dddd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:25.239019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db814e4-5802-4764-b121-2451c13dddd9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:25.239298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db814e4-5802-4764-b121-2451c13dddd9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:25.239597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db814e4-5802-4764-b121-2451c13dddd9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:25.240766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db814e4-5802-4764-b121-2451c13dddd9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:25.242227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db814e4-5802-4764-b121-2451c13dddd9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:25.243851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 10/19] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:53:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:25.249500 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db814e4-5802-4764-b121-2451c13dddd9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:25.271480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57034a6-220c-42ad-95ca-2d4af7ccf583 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:25.305719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7797cd-0f79-4e56-a68a-9ea561586e80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:25.306338 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7797cd-0f79-4e56-a68a-9ea561586e80 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:25.306622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7797cd-0f79-4e56-a68a-9ea561586e80 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:25.306897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7797cd-0f79-4e56-a68a-9ea561586e80 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:25.307623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7797cd-0f79-4e56-a68a-9ea561586e80 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:25.308161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7797cd-0f79-4e56-a68a-9ea561586e80 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:25.309299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7797cd-0f79-4e56-a68a-9ea561586e80 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:25.327431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 10/20] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:53:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:25.331282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:25.354170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db814e4-5802-4764-b121-2451c13dddd9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:25.391563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b581f2c7-b26a-4ac8-9309-8c46ee9a02ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:25.392205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b581f2c7-b26a-4ac8-9309-8c46ee9a02ba None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:25.392490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b581f2c7-b26a-4ac8-9309-8c46ee9a02ba None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:25.392771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b581f2c7-b26a-4ac8-9309-8c46ee9a02ba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:25.393583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b581f2c7-b26a-4ac8-9309-8c46ee9a02ba None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.domain.id=default, user.id=e711c7fc76434a0b86b92dc8bcf69e87) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:25.393935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b581f2c7-b26a-4ac8-9309-8c46ee9a02ba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:25.395504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b581f2c7-b26a-4ac8-9309-8c46ee9a02ba None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:25.403141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 11/21] 10.222.0.42 () {56 vars in 1322 bytes} [Sun Jul 31 00:53:25 2022] GET /identity/v3/role_assignments?user.id=e711c7fc76434a0b86b92dc8bcf69e87&scope.domain.id=default&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 233 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:27.774989 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68961c7c-c57e-4b3d-a93a-f13dba613d3c None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:27.775509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68961c7c-c57e-4b3d-a93a-f13dba613d3c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:27.775605 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68961c7c-c57e-4b3d-a93a-f13dba613d3c None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:27.776906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 11/22] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:27.782357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1575b3-c750-4b90-89c8-12ad51072bb8 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:27.783216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1575b3-c750-4b90-89c8-12ad51072bb8 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:27.783750 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1575b3-c750-4b90-89c8-12ad51072bb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:27.810300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-7f1575b3-c750-4b90-89c8-12ad51072bb8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:27.811855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7f1575b3-c750-4b90-89c8-12ad51072bb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:27.902813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 12/23] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:27.940919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68961c7c-c57e-4b3d-a93a-f13dba613d3c None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:28.077420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68961c7c-c57e-4b3d-a93a-f13dba613d3c None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:28.107762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:28.134625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3bc63a-e2e9-48af-b60d-2e9914b8b41c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:28.135297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3bc63a-e2e9-48af-b60d-2e9914b8b41c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:28.135606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3bc63a-e2e9-48af-b60d-2e9914b8b41c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:28.135877 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3bc63a-e2e9-48af-b60d-2e9914b8b41c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:28.148187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3bc63a-e2e9-48af-b60d-2e9914b8b41c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:28.149705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3bc63a-e2e9-48af-b60d-2e9914b8b41c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:28.155492 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9339d72-348d-441c-b8b6-a85d05fab68f None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:28.155785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9339d72-348d-441c-b8b6-a85d05fab68f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:28.156092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9339d72-348d-441c-b8b6-a85d05fab68f None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:28.157692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 13/24] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:28.163715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0f3a71-9359-4432-ab5d-90610d7bffc1 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:28.164036 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0f3a71-9359-4432-ab5d-90610d7bffc1 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:28.164272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0f3a71-9359-4432-ab5d-90610d7bffc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-fb3bc63a-e2e9-48af-b60d-2e9914b8b41c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:53:28.192701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:28.193576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:28.194401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:53:28.194401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:53:28.194401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:53:28.194401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:53:28.194401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 12/25] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:53:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:28.201748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ad0f3a71-9359-4432-ab5d-90610d7bffc1 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:28.203027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ad0f3a71-9359-4432-ab5d-90610d7bffc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:28.207145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3bc63a-e2e9-48af-b60d-2e9914b8b41c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:28.366796 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 14/26] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:53:28.400185 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0f3a71-9359-4432-ab5d-90610d7bffc1 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:28.403692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:28.404534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3bc63a-e2e9-48af-b60d-2e9914b8b41c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:28.533270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb773c86-b98e-4ee6-9364-3ee85794a775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:28.534196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb773c86-b98e-4ee6-9364-3ee85794a775 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:28.534658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb773c86-b98e-4ee6-9364-3ee85794a775 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:28.534928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb773c86-b98e-4ee6-9364-3ee85794a775 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:28.535692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb773c86-b98e-4ee6-9364-3ee85794a775 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:28.535990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb773c86-b98e-4ee6-9364-3ee85794a775 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:28.537426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb773c86-b98e-4ee6-9364-3ee85794a775 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:28.537891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0f3a71-9359-4432-ab5d-90610d7bffc1 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:28.565102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 13/27] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:53:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 359 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:28.566731 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e6d51c-a6a6-41e7-8825-fa5049027c12 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:28.567019 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e6d51c-a6a6-41e7-8825-fa5049027c12 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:28.567290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e6d51c-a6a6-41e7-8825-fa5049027c12 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:28.568785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 14/28] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:28.580019 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e6d51c-a6a6-41e7-8825-fa5049027c12 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:28.620324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37650944-202f-4312-94e1-3e6fd8f1ad3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:28.620930 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37650944-202f-4312-94e1-3e6fd8f1ad3c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:28.621196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37650944-202f-4312-94e1-3e6fd8f1ad3c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:28.621470 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37650944-202f-4312-94e1-3e6fd8f1ad3c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:28.636208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37650944-202f-4312-94e1-3e6fd8f1ad3c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-37650944-202f-4312-94e1-3e6fd8f1ad3c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:28.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:28.642944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:28.644650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:28.644650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:28.644650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:28.644650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:28.644650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:28.644650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:28.644650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:28.644650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:28.644650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:28.644650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:28.644650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:28.644650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:53:28.646809 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 15/29] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:53:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:28.649818 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd812e8-dce5-4080-9a25-ed65787aceac None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:28.650338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd812e8-dce5-4080-9a25-ed65787aceac None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:28.650817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd812e8-dce5-4080-9a25-ed65787aceac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:28.670582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-bbd812e8-dce5-4080-9a25-ed65787aceac None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:28.672231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-bbd812e8-dce5-4080-9a25-ed65787aceac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:28.747430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:28.759482 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 16/30] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:28.763515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad369fd-0ea6-448f-9efa-73d7e0f337cb None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:28.764057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad369fd-0ea6-448f-9efa-73d7e0f337cb None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:28.764552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad369fd-0ea6-448f-9efa-73d7e0f337cb None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:28.766423 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 17/31] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:28 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:28.772199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad369fd-0ea6-448f-9efa-73d7e0f337cb None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:28.774503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:28.860115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e6d51c-a6a6-41e7-8825-fa5049027c12 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:28.902020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88543e57-b1c4-4f62-80b3-969a639b2aa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:28.902715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88543e57-b1c4-4f62-80b3-969a639b2aa0 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:28.903050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88543e57-b1c4-4f62-80b3-969a639b2aa0 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:28.903331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88543e57-b1c4-4f62-80b3-969a639b2aa0 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:28.904254 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88543e57-b1c4-4f62-80b3-969a639b2aa0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:28.905720 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88543e57-b1c4-4f62-80b3-969a639b2aa0 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:28.927111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad369fd-0ea6-448f-9efa-73d7e0f337cb None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:28.965373 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280b1642-c945-4e01-88a3-b61ab2a7efe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:28.966333 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280b1642-c945-4e01-88a3-b61ab2a7efe1 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:28.966812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280b1642-c945-4e01-88a3-b61ab2a7efe1 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:28.968012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280b1642-c945-4e01-88a3-b61ab2a7efe1 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:28.969317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280b1642-c945-4e01-88a3-b61ab2a7efe1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:28.969634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280b1642-c945-4e01-88a3-b61ab2a7efe1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:28.971024 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280b1642-c945-4e01-88a3-b61ab2a7efe1 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:28.980391 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 18/32] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:28.985441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb7172d-7e6f-4d6f-a50c-21379cc684f7 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:28.985441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb7172d-7e6f-4d6f-a50c-21379cc684f7 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:28.985441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb7172d-7e6f-4d6f-a50c-21379cc684f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.000607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-8fb7172d-7e6f-4d6f-a50c-21379cc684f7 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:29.001783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8fb7172d-7e6f-4d6f-a50c-21379cc684f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:29.055292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:29.059413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 15/33] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:53:28 2022] POST /identity/v3/projects => generated 301 bytes in 477 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:53:29.059413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88543e57-b1c4-4f62-80b3-969a639b2aa0 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:29.086250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 19/34] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:29.087982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8be0d5-c81d-4906-bea6-cc5edf52dea4 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:29.088425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8be0d5-c81d-4906-bea6-cc5edf52dea4 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:29.088866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8be0d5-c81d-4906-bea6-cc5edf52dea4 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.090389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 20/35] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:29.095451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8be0d5-c81d-4906-bea6-cc5edf52dea4 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:29.199859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:29.211979 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88543e57-b1c4-4f62-80b3-969a639b2aa0 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:29.232768 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8be0d5-c81d-4906-bea6-cc5edf52dea4 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:29.256196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acd7437-cb75-481c-80fe-b9c1beac71d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:29.256829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acd7437-cb75-481c-80fe-b9c1beac71d1 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:29.257105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acd7437-cb75-481c-80fe-b9c1beac71d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:29.257382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acd7437-cb75-481c-80fe-b9c1beac71d1 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.258096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acd7437-cb75-481c-80fe-b9c1beac71d1 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:29.259664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acd7437-cb75-481c-80fe-b9c1beac71d1 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:29.269398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a34e29-d95b-49fa-9476-4e72a7b43469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:29.270087 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a34e29-d95b-49fa-9476-4e72a7b43469 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:29.270367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a34e29-d95b-49fa-9476-4e72a7b43469 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:29.270685 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a34e29-d95b-49fa-9476-4e72a7b43469 None admin] PATH_INFO: `/v3/users/e711c7fc76434a0b86b92dc8bcf69e87` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.271938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a34e29-d95b-49fa-9476-4e72a7b43469 None admin] RBAC: Authorizing `identity:get_user(user_id=e711c7fc76434a0b86b92dc8bcf69e87)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:29.273379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a34e29-d95b-49fa-9476-4e72a7b43469 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:29.274976 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 21/36] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:29 2022] GET /identity/v3/users/e711c7fc76434a0b86b92dc8bcf69e87 => generated 247 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:29.276113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80d13ae-c64f-4888-b7ad-fd5243bdd7ff None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:29.276440 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80d13ae-c64f-4888-b7ad-fd5243bdd7ff None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:29.276788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80d13ae-c64f-4888-b7ad-fd5243bdd7ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.283068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 16/37] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Jul 31 00:53:29 2022] POST /identity/v3/roles => generated 219 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:53:29.283455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acd7437-cb75-481c-80fe-b9c1beac71d1 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:29.299152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c80d13ae-c64f-4888-b7ad-fd5243bdd7ff None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:29.300963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c80d13ae-c64f-4888-b7ad-fd5243bdd7ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:29.362229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:29.389799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 22/38] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:29.389799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08223eed-6d01-4820-b5c8-500ab96bca5d None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:29.389799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08223eed-6d01-4820-b5c8-500ab96bca5d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:29.389799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08223eed-6d01-4820-b5c8-500ab96bca5d None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.389799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 23/39] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:29.389799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08223eed-6d01-4820-b5c8-500ab96bca5d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:29.415790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acd7437-cb75-481c-80fe-b9c1beac71d1 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:29.453100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62439c5-5818-493a-96dd-45eec22388bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:29.453647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62439c5-5818-493a-96dd-45eec22388bf None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:29.453943 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62439c5-5818-493a-96dd-45eec22388bf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:29.454609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62439c5-5818-493a-96dd-45eec22388bf None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.454935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62439c5-5818-493a-96dd-45eec22388bf None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:29.456895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62439c5-5818-493a-96dd-45eec22388bf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:29.466948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:29.475049 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 17/40] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Jul 31 00:53:29 2022] POST /identity/v3/roles => generated 225 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:53:29.477810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a9c91-00a9-4201-9b21-f6796827c7c0 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:29.477810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a9c91-00a9-4201-9b21-f6796827c7c0 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:29.477810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a9c91-00a9-4201-9b21-f6796827c7c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.499376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-558a9c91-00a9-4201-9b21-f6796827c7c0 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:29.501438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-558a9c91-00a9-4201-9b21-f6796827c7c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:29.523976 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08223eed-6d01-4820-b5c8-500ab96bca5d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:29.575131 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c1de63-063e-417f-ad1c-ab2622d8d057 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:29.575830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c1de63-063e-417f-ad1c-ab2622d8d057 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:29.576104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c1de63-063e-417f-ad1c-ab2622d8d057 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:29.576380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c1de63-063e-417f-ad1c-ab2622d8d057 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.578068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c1de63-063e-417f-ad1c-ab2622d8d057 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:29.579629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c1de63-063e-417f-ad1c-ab2622d8d057 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:29.586795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 24/41] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:53:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:29.586795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c1de63-063e-417f-ad1c-ab2622d8d057 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:29.603445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 18/42] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:29.603445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558a9c91-00a9-4201-9b21-f6796827c7c0 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:29.655928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:29.710712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c1de63-063e-417f-ad1c-ab2622d8d057 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:29.715759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558a9c91-00a9-4201-9b21-f6796827c7c0 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:29.744467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:29.744769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2282789-79ea-497b-b5da-4b7e602a47a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:29.745458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2282789-79ea-497b-b5da-4b7e602a47a9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:29.746093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2282789-79ea-497b-b5da-4b7e602a47a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:29.746407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2282789-79ea-497b-b5da-4b7e602a47a9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.747625 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2282789-79ea-497b-b5da-4b7e602a47a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:29.749070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2282789-79ea-497b-b5da-4b7e602a47a9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:29.750618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d03dd76-9ba4-4695-98b6-85cc803468a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:29.750755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 25/43] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:53:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:29.751293 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d03dd76-9ba4-4695-98b6-85cc803468a1 None admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:29.751497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2282789-79ea-497b-b5da-4b7e602a47a9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:29.751655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d03dd76-9ba4-4695-98b6-85cc803468a1 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:29.751957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d03dd76-9ba4-4695-98b6-85cc803468a1 None admin] PATH_INFO: `/v3/domains/default/users/e711c7fc76434a0b86b92dc8bcf69e87/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.757135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d03dd76-9ba4-4695-98b6-85cc803468a1 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=e711c7fc76434a0b86b92dc8bcf69e87, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:29.758581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d03dd76-9ba4-4695-98b6-85cc803468a1 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:29.773973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 19/44] 10.222.0.42 () {58 vars in 1361 bytes} [Sun Jul 31 00:53:29 2022] PUT /identity/v3/domains/default/users/e711c7fc76434a0b86b92dc8bcf69e87/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:53:29.774755 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d03dd76-9ba4-4695-98b6-85cc803468a1 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:29.891001 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d03dd76-9ba4-4695-98b6-85cc803468a1 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:29.893113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2282789-79ea-497b-b5da-4b7e602a47a9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:29.916221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:29.934924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d80ea77-d27f-40c5-931b-58f5e4e605e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:29.935596 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d80ea77-d27f-40c5-931b-58f5e4e605e8 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:29.936486 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d80ea77-d27f-40c5-931b-58f5e4e605e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:29.936587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a583aa2-b61c-49b8-bdb4-85f3e577213f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:29.937010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d80ea77-d27f-40c5-931b-58f5e4e605e8 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.937084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a583aa2-b61c-49b8-bdb4-85f3e577213f None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:29.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d80ea77-d27f-40c5-931b-58f5e4e605e8 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:29.937218 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a583aa2-b61c-49b8-bdb4-85f3e577213f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:29.937429 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a583aa2-b61c-49b8-bdb4-85f3e577213f None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:29.939723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a583aa2-b61c-49b8-bdb4-85f3e577213f None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:29.939826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d80ea77-d27f-40c5-931b-58f5e4e605e8 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:29.939911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a583aa2-b61c-49b8-bdb4-85f3e577213f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:29.967509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 20/45] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:53:29 2022] POST /identity/v3/projects => generated 312 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:53:30.067214 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 26/46] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Jul 31 00:53:29 2022] POST /identity/v3/roles => generated 223 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:53:31.784963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f7ef8a-eb21-4de4-ab17-55910153f204 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:31.785483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f7ef8a-eb21-4de4-ab17-55910153f204 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:31.785921 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f7ef8a-eb21-4de4-ab17-55910153f204 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:31.788468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 21/47] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:31 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:31.796553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784c7f72-b3e0-4988-b8d8-b67723fef1ff None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:31.797358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784c7f72-b3e0-4988-b8d8-b67723fef1ff None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:31.797503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784c7f72-b3e0-4988-b8d8-b67723fef1ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:31.820644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-784c7f72-b3e0-4988-b8d8-b67723fef1ff None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:31.822260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-784c7f72-b3e0-4988-b8d8-b67723fef1ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:31.897437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 27/48] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:31.921487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f7ef8a-eb21-4de4-ab17-55910153f204 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:31.976786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:32.013836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e13f5fb-770e-4e1c-95d7-0cd85f5f5872 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:32.014174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e13f5fb-770e-4e1c-95d7-0cd85f5f5872 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:32.014396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e13f5fb-770e-4e1c-95d7-0cd85f5f5872 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:32.015866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 28/49] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:53:32.024354 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bafe76-01b4-44de-b342-582aa9ec9cf8 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:32.024758 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bafe76-01b4-44de-b342-582aa9ec9cf8 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:32.025151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bafe76-01b4-44de-b342-582aa9ec9cf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:32.052956 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-58bafe76-01b4-44de-b342-582aa9ec9cf8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:32.054590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f7ef8a-eb21-4de4-ab17-55910153f204 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:32.054828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-58bafe76-01b4-44de-b342-582aa9ec9cf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:32.103052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208d25fb-34fc-47c5-bd66-2fd87a69453b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:32.104110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208d25fb-34fc-47c5-bd66-2fd87a69453b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:32.104612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208d25fb-34fc-47c5-bd66-2fd87a69453b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:32.105110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208d25fb-34fc-47c5-bd66-2fd87a69453b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:32.121410 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208d25fb-34fc-47c5-bd66-2fd87a69453b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-208d25fb-34fc-47c5-bd66-2fd87a69453b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:32.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:32.130628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:53:32.132590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 22/50] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:53:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:32.136806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208d25fb-34fc-47c5-bd66-2fd87a69453b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:32.160753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 29/51] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:53:32.184381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bafe76-01b4-44de-b342-582aa9ec9cf8 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:32.230503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:32.269470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208d25fb-34fc-47c5-bd66-2fd87a69453b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:32.272533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:32.298280 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bafe76-01b4-44de-b342-582aa9ec9cf8 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:32.328131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a9b8f3-da90-4913-a1e8-ce6255096475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:32.328766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a9b8f3-da90-4913-a1e8-ce6255096475 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:32.329068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a9b8f3-da90-4913-a1e8-ce6255096475 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:32.329343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a9b8f3-da90-4913-a1e8-ce6255096475 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:32.330071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a9b8f3-da90-4913-a1e8-ce6255096475 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:32.330392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a9b8f3-da90-4913-a1e8-ce6255096475 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:32.331856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a9b8f3-da90-4913-a1e8-ce6255096475 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:32.342561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225c9e17-734b-49ab-b9d1-144072580417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:32.343713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225c9e17-734b-49ab-b9d1-144072580417 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:32.344214 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225c9e17-734b-49ab-b9d1-144072580417 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:32.344706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225c9e17-734b-49ab-b9d1-144072580417 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:32.345037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 23/52] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:32.347323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225c9e17-734b-49ab-b9d1-144072580417 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:32.350178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225c9e17-734b-49ab-b9d1-144072580417 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:32.351373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a9b8f3-da90-4913-a1e8-ce6255096475 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:32.353373 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 30/53] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:53:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:32.360453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225c9e17-734b-49ab-b9d1-144072580417 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:32.475494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a9b8f3-da90-4913-a1e8-ce6255096475 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:32.513198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225c9e17-734b-49ab-b9d1-144072580417 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:32.527647 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:32.537356 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f69fd1-6798-4a72-8e3b-1f025844e390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:32.538384 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f69fd1-6798-4a72-8e3b-1f025844e390 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:32.538888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f69fd1-6798-4a72-8e3b-1f025844e390 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:32.539405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f69fd1-6798-4a72-8e3b-1f025844e390 None admin] PATH_INFO: `/v3/users/e711c7fc76434a0b86b92dc8bcf69e87` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:32.541497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f69fd1-6798-4a72-8e3b-1f025844e390 None admin] RBAC: Authorizing `identity:get_user(user_id=e711c7fc76434a0b86b92dc8bcf69e87)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:32.544142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f69fd1-6798-4a72-8e3b-1f025844e390 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:32.547071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 24/54] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:32 2022] GET /identity/v3/users/e711c7fc76434a0b86b92dc8bcf69e87 => generated 247 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:32.554095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f69fd1-6798-4a72-8e3b-1f025844e390 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:32.562659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443dbd78-a06e-40b7-ae8f-072d99b0eada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:32.563487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443dbd78-a06e-40b7-ae8f-072d99b0eada None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:32.563852 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443dbd78-a06e-40b7-ae8f-072d99b0eada None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:32.564210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443dbd78-a06e-40b7-ae8f-072d99b0eada None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:32.564995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443dbd78-a06e-40b7-ae8f-072d99b0eada None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:32.566715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443dbd78-a06e-40b7-ae8f-072d99b0eada None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:32.602266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 31/55] 10.222.0.42 () {60 vars in 1140 bytes} [Sun Jul 31 00:53:32 2022] POST /identity/v3/projects => generated 298 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:53:32.605104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:32.690820 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f69fd1-6798-4a72-8e3b-1f025844e390 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:32.735153 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f062b0-80bb-48bc-b910-3536ee0d9a9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:32.736030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f062b0-80bb-48bc-b910-3536ee0d9a9f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:32.736433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f062b0-80bb-48bc-b910-3536ee0d9a9f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:32.736821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f062b0-80bb-48bc-b910-3536ee0d9a9f None admin] PATH_INFO: `/v3/domains/default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:32.738449 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f062b0-80bb-48bc-b910-3536ee0d9a9f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:32.740292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f062b0-80bb-48bc-b910-3536ee0d9a9f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:32.742360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 25/56] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:53:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:32.749744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443dbd78-a06e-40b7-ae8f-072d99b0eada None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:32.930334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443dbd78-a06e-40b7-ae8f-072d99b0eada None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:32.991701 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55156dc-b9ec-4b17-9332-c7082dbddfc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:32.992657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55156dc-b9ec-4b17-9332-c7082dbddfc6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:32.993076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55156dc-b9ec-4b17-9332-c7082dbddfc6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:32.993503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55156dc-b9ec-4b17-9332-c7082dbddfc6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:32.994829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55156dc-b9ec-4b17-9332-c7082dbddfc6 None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.domain.id=default, user.id=e711c7fc76434a0b86b92dc8bcf69e87) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:32.995365 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55156dc-b9ec-4b17-9332-c7082dbddfc6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:32.998043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55156dc-b9ec-4b17-9332-c7082dbddfc6 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:33.008021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 32/57] 10.222.0.42 () {56 vars in 1322 bytes} [Sun Jul 31 00:53:32 2022] GET /identity/v3/role_assignments?user.id=e711c7fc76434a0b86b92dc8bcf69e87&scope.domain.id=default&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 530 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:34.736810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f19a65d-333b-476e-b810-e805a89729e1 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:34.737878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f19a65d-333b-476e-b810-e805a89729e1 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:34.738067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f19a65d-333b-476e-b810-e805a89729e1 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:34.739929 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 26/58] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:34.745876 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7cccbb-efab-444a-a4e2-f9bb279e2ca5 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:34.746427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7cccbb-efab-444a-a4e2-f9bb279e2ca5 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:34.746513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7cccbb-efab-444a-a4e2-f9bb279e2ca5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:34.766533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-8b7cccbb-efab-444a-a4e2-f9bb279e2ca5 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:34.767905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8b7cccbb-efab-444a-a4e2-f9bb279e2ca5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:34.787630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:34.849917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 33/59] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:34.875647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f19a65d-333b-476e-b810-e805a89729e1 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:35.006054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:35.011594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f19a65d-333b-476e-b810-e805a89729e1 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:35.050582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3715c707-d6d0-4f55-a18f-1c7fbd52e3cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:35.051537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3715c707-d6d0-4f55-a18f-1c7fbd52e3cf None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:35.051822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3715c707-d6d0-4f55-a18f-1c7fbd52e3cf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:35.052096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3715c707-d6d0-4f55-a18f-1c7fbd52e3cf None admin] PATH_INFO: `/v3/domains/default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:35.053288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3715c707-d6d0-4f55-a18f-1c7fbd52e3cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:35.054774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3715c707-d6d0-4f55-a18f-1c7fbd52e3cf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:35.056455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 27/60] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:53:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:35.062609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7cccbb-efab-444a-a4e2-f9bb279e2ca5 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:35.184820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7cccbb-efab-444a-a4e2-f9bb279e2ca5 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:35.228858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ba90c5-5688-45e7-818d-c3d6f0aca1e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:35.229508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ba90c5-5688-45e7-818d-c3d6f0aca1e5 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:35.229791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ba90c5-5688-45e7-818d-c3d6f0aca1e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:35.230071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ba90c5-5688-45e7-818d-c3d6f0aca1e5 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:35.230780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ba90c5-5688-45e7-818d-c3d6f0aca1e5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:35.232338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ba90c5-5688-45e7-818d-c3d6f0aca1e5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:35.264870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 34/61] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:53:35 2022] POST /identity/v3/users => generated 275 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:53:39.271763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc43ad-fcf2-4aa3-b876-6c0d1b3076d5 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:39.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc43ad-fcf2-4aa3-b876-6c0d1b3076d5 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:39.273413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc43ad-fcf2-4aa3-b876-6c0d1b3076d5 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:39.275415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 28/62] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:39.280673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f51207-fcd6-4749-8073-ccb826cd80cc None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:39.281271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d83487-8272-4687-baf6-e9834719bebb None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:39.281467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d83487-8272-4687-baf6-e9834719bebb None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:39.281749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d83487-8272-4687-baf6-e9834719bebb None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:39.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f51207-fcd6-4749-8073-ccb826cd80cc None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:39.282611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f51207-fcd6-4749-8073-ccb826cd80cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:39.283152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 29/63] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:53:39.289552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29daccc-2a0a-45e9-9bf9-4f3fca4b9dc9 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:39.290148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29daccc-2a0a-45e9-9bf9-4f3fca4b9dc9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:39.290624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29daccc-2a0a-45e9-9bf9-4f3fca4b9dc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:39.322837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-10f51207-fcd6-4749-8073-ccb826cd80cc None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:39.324146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-10f51207-fcd6-4749-8073-ccb826cd80cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:39.327419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f29daccc-2a0a-45e9-9bf9-4f3fca4b9dc9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:39.328689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f29daccc-2a0a-45e9-9bf9-4f3fca4b9dc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:39.334713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:39.469632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 35/64] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:39.481650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 30/65] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:39.493526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a842e3d-8495-4c7d-bfa2-a8a0849530ed None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:39.493795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a842e3d-8495-4c7d-bfa2-a8a0849530ed None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:39.494070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a842e3d-8495-4c7d-bfa2-a8a0849530ed None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:39.495623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 36/66] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:53:39.501259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d68ab6d-40b3-43de-853a-92aa2f55f54b None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:39.501552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d68ab6d-40b3-43de-853a-92aa2f55f54b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:39.501871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d68ab6d-40b3-43de-853a-92aa2f55f54b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:39.516613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a842e3d-8495-4c7d-bfa2-a8a0849530ed None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:39.538460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-5d68ab6d-40b3-43de-853a-92aa2f55f54b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:39.541182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5d68ab6d-40b3-43de-853a-92aa2f55f54b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:39.687960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 31/67] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:53:39.689754 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d68ab6d-40b3-43de-853a-92aa2f55f54b None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:39.716682 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a842e3d-8495-4c7d-bfa2-a8a0849530ed None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:39.761054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:39.768200 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0187033a-6e61-400b-adca-5862d4060c82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:39.769493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0187033a-6e61-400b-adca-5862d4060c82 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:39.770055 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0187033a-6e61-400b-adca-5862d4060c82 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:39.770209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0187033a-6e61-400b-adca-5862d4060c82 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:39.780189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:39.784911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0187033a-6e61-400b-adca-5862d4060c82 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-0187033a-6e61-400b-adca-5862d4060c82 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:39.795545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:39.796515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:39.797253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:39.797253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:39.797253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:39.797253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:39.797253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:39.797253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:39.797253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:39.797253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:39.797253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:39.797253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:39.797253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:53:39.797253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:53:39.798293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 37/68] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:53:39.798891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0187033a-6e61-400b-adca-5862d4060c82 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:39.855594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d68ab6d-40b3-43de-853a-92aa2f55f54b None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:39.894943 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bcc25a-1fd1-4bc6-9163-f0cf6ad68a16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:39.895798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bcc25a-1fd1-4bc6-9163-f0cf6ad68a16 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:39.896074 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bcc25a-1fd1-4bc6-9163-f0cf6ad68a16 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:39.896348 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bcc25a-1fd1-4bc6-9163-f0cf6ad68a16 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:39.907926 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bcc25a-1fd1-4bc6-9163-f0cf6ad68a16 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-47bcc25a-1fd1-4bc6-9163-f0cf6ad68a16 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:39.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:39.914662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:53:39.915467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 32/69] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:53:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:39.918754 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d0e8b3-6663-42cf-b5a6-c4ac7b4caa02 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:39.918754 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d0e8b3-6663-42cf-b5a6-c4ac7b4caa02 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:39.918754 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d0e8b3-6663-42cf-b5a6-c4ac7b4caa02 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:39.919460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 33/70] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:39.920441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8eee77-54fa-4d7b-bcf4-6ca579b645ea None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:39.920674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8eee77-54fa-4d7b-bcf4-6ca579b645ea None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:39.921575 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8eee77-54fa-4d7b-bcf4-6ca579b645ea None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:39.922989 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 34/71] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:39.923751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8eee77-54fa-4d7b-bcf4-6ca579b645ea None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:39.931684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0187033a-6e61-400b-adca-5862d4060c82 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:39.972570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529369f3-5952-44a9-8cbf-9958c6af663c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:39.973223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529369f3-5952-44a9-8cbf-9958c6af663c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:39.973508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529369f3-5952-44a9-8cbf-9958c6af663c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:39.973811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529369f3-5952-44a9-8cbf-9958c6af663c None admin] PATH_INFO: `/v3/roles/member` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:39.986356 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529369f3-5952-44a9-8cbf-9958c6af663c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-529369f3-5952-44a9-8cbf-9958c6af663c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:39.991433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:39.992522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:53:39.993340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 38/72] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:39.994529 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529369f3-5952-44a9-8cbf-9958c6af663c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:40.052323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:40.069218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8eee77-54fa-4d7b-bcf4-6ca579b645ea None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:40.076206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:40.121870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e8f2f2-bc0b-4adc-b377-05bc04b9866b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:40.122517 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e8f2f2-bc0b-4adc-b377-05bc04b9866b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:40.122803 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e8f2f2-bc0b-4adc-b377-05bc04b9866b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:40.123072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e8f2f2-bc0b-4adc-b377-05bc04b9866b None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:40.123870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e8f2f2-bc0b-4adc-b377-05bc04b9866b None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:40.124193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e8f2f2-bc0b-4adc-b377-05bc04b9866b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:40.125771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e8f2f2-bc0b-4adc-b377-05bc04b9866b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:40.130009 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529369f3-5952-44a9-8cbf-9958c6af663c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:40.136777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 35/73] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:53:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:40.138102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8763a6fe-231d-4564-a924-f5c777e4f748 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:40.138464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8763a6fe-231d-4564-a924-f5c777e4f748 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:40.138665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8763a6fe-231d-4564-a924-f5c777e4f748 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:40.165453 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-8763a6fe-231d-4564-a924-f5c777e4f748 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:40.167037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8763a6fe-231d-4564-a924-f5c777e4f748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:40.183116 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af354796-38d9-49f5-824e-e3ced1edd054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:40.184208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af354796-38d9-49f5-824e-e3ced1edd054 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:40.184700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af354796-38d9-49f5-824e-e3ced1edd054 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:40.185191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af354796-38d9-49f5-824e-e3ced1edd054 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:40.186546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af354796-38d9-49f5-824e-e3ced1edd054 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:40.187064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af354796-38d9-49f5-824e-e3ced1edd054 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:40.189706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af354796-38d9-49f5-824e-e3ced1edd054 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:40.204355 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 39/74] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:40.209972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3829b863-d098-401f-8852-6391b7e10422 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:40.210454 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3829b863-d098-401f-8852-6391b7e10422 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:40.210947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3829b863-d098-401f-8852-6391b7e10422 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:40.241216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-3829b863-d098-401f-8852-6391b7e10422 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:40.243216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-3829b863-d098-401f-8852-6391b7e10422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:40.266104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 36/75] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:40.268256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8763a6fe-231d-4564-a924-f5c777e4f748 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:40.356369 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 40/76] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:40.357584 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3829b863-d098-401f-8852-6391b7e10422 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:40.400712 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:40.428482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8763a6fe-231d-4564-a924-f5c777e4f748 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:40.437404 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:40.490101 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e5acd0-a182-443e-b06e-3995c7f85e26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:40.491092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e5acd0-a182-443e-b06e-3995c7f85e26 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:40.491564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e5acd0-a182-443e-b06e-3995c7f85e26 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:40.492014 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e5acd0-a182-443e-b06e-3995c7f85e26 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:40.493259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e5acd0-a182-443e-b06e-3995c7f85e26 None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:40.493776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e5acd0-a182-443e-b06e-3995c7f85e26 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:40.496267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e5acd0-a182-443e-b06e-3995c7f85e26 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:40.517657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 37/77] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:53:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:40.517657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e5acd0-a182-443e-b06e-3995c7f85e26 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:40.524556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3829b863-d098-401f-8852-6391b7e10422 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:40.571571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62366f4c-fba4-40fd-83d9-078554a454a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:40.572246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62366f4c-fba4-40fd-83d9-078554a454a7 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:40.572531 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62366f4c-fba4-40fd-83d9-078554a454a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:40.572810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62366f4c-fba4-40fd-83d9-078554a454a7 None admin] PATH_INFO: `/v3/users/a614cb9f929748ecb70ea9ce0aed9916` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:40.589198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62366f4c-fba4-40fd-83d9-078554a454a7 None admin] RBAC: Authorizing `identity:get_user(user_id=a614cb9f929748ecb70ea9ce0aed9916)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:40.590731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62366f4c-fba4-40fd-83d9-078554a454a7 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:40.592569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 41/78] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:40 2022] GET /identity/v3/users/a614cb9f929748ecb70ea9ce0aed9916 => generated 275 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:40.593380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62366f4c-fba4-40fd-83d9-078554a454a7 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:40.659970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e5acd0-a182-443e-b06e-3995c7f85e26 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:40.700499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ab118c-672a-4958-8762-59d3bc84c55d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:40.701157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ab118c-672a-4958-8762-59d3bc84c55d None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:40.701441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ab118c-672a-4958-8762-59d3bc84c55d None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:40.701718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ab118c-672a-4958-8762-59d3bc84c55d None admin] PATH_INFO: `/v3/users/e711c7fc76434a0b86b92dc8bcf69e87` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:40.702975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ab118c-672a-4958-8762-59d3bc84c55d None admin] RBAC: Authorizing `identity:get_user(user_id=e711c7fc76434a0b86b92dc8bcf69e87)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:40.704506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ab118c-672a-4958-8762-59d3bc84c55d None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:40.706753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 38/79] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:40 2022] GET /identity/v3/users/e711c7fc76434a0b86b92dc8bcf69e87 => generated 247 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:40.707612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ab118c-672a-4958-8762-59d3bc84c55d None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:40.722386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:40.736787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62366f4c-fba4-40fd-83d9-078554a454a7 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:40.750863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:40.785206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb636bc4-5b96-44e8-b861-4233563c8c04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:40.785919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb636bc4-5b96-44e8-b861-4233563c8c04 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:40.786188 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb636bc4-5b96-44e8-b861-4233563c8c04 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:40.786457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb636bc4-5b96-44e8-b861-4233563c8c04 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:40.787726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb636bc4-5b96-44e8-b861-4233563c8c04 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:40.789474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb636bc4-5b96-44e8-b861-4233563c8c04 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:40.791304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 42/80] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:53:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:40.792119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb636bc4-5b96-44e8-b861-4233563c8c04 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:40.819012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ab118c-672a-4958-8762-59d3bc84c55d None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:40.857602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abea4c2a-317f-49ec-a59b-3ef2f80adc7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:40.858688 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abea4c2a-317f-49ec-a59b-3ef2f80adc7c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:40.859185 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abea4c2a-317f-49ec-a59b-3ef2f80adc7c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:40.859573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abea4c2a-317f-49ec-a59b-3ef2f80adc7c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:40.876533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abea4c2a-317f-49ec-a59b-3ef2f80adc7c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-abea4c2a-317f-49ec-a59b-3ef2f80adc7c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:40.886450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:40.887554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:53:40.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 39/81] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:53:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:40.889797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abea4c2a-317f-49ec-a59b-3ef2f80adc7c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:40.916241 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb636bc4-5b96-44e8-b861-4233563c8c04 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:40.957966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211b8e8-e1c0-4517-b8a6-ffb74a5351da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:40.958726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211b8e8-e1c0-4517-b8a6-ffb74a5351da None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:40.959022 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211b8e8-e1c0-4517-b8a6-ffb74a5351da None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:40.959302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211b8e8-e1c0-4517-b8a6-ffb74a5351da None admin] PATH_INFO: `/v3/users/a614cb9f929748ecb70ea9ce0aed9916` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:40.960671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b211b8e8-e1c0-4517-b8a6-ffb74a5351da None admin] RBAC: Authorizing `identity:get_user(user_id=a614cb9f929748ecb70ea9ce0aed9916)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:40.962386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b211b8e8-e1c0-4517-b8a6-ffb74a5351da None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:40.964230 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 43/82] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:40 2022] GET /identity/v3/users/a614cb9f929748ecb70ea9ce0aed9916 => generated 275 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:40.965137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211b8e8-e1c0-4517-b8a6-ffb74a5351da None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:40.991817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:41.017837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:41.037763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abea4c2a-317f-49ec-a59b-3ef2f80adc7c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:41.081102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e86d1b4-2499-4fb9-879e-8cb6860fe874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:41.081936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e86d1b4-2499-4fb9-879e-8cb6860fe874 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:41.082274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e86d1b4-2499-4fb9-879e-8cb6860fe874 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:41.082739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e86d1b4-2499-4fb9-879e-8cb6860fe874 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:41.089842 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211b8e8-e1c0-4517-b8a6-ffb74a5351da None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:41.140534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21d5c6e-a0dc-4c7d-97cd-38c377a0636f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:41.141407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21d5c6e-a0dc-4c7d-97cd-38c377a0636f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:41.141790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21d5c6e-a0dc-4c7d-97cd-38c377a0636f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:41.142345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21d5c6e-a0dc-4c7d-97cd-38c377a0636f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:41.165585 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e86d1b4-2499-4fb9-879e-8cb6860fe874 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:41.167279 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e86d1b4-2499-4fb9-879e-8cb6860fe874 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:41.231037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21d5c6e-a0dc-4c7d-97cd-38c377a0636f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:41.232620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21d5c6e-a0dc-4c7d-97cd-38c377a0636f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:41.285385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:41.287828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:41.326345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 40/83] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:53:41.327755 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e86d1b4-2499-4fb9-879e-8cb6860fe874 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:41.370736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 44/84] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:53:41.371992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21d5c6e-a0dc-4c7d-97cd-38c377a0636f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:41.443849 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e86d1b4-2499-4fb9-879e-8cb6860fe874 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:41.486088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c48678-3987-4533-8f23-09a7e35718aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:41.486760 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c48678-3987-4533-8f23-09a7e35718aa None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:41.487048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c48678-3987-4533-8f23-09a7e35718aa None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:41.487457 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c48678-3987-4533-8f23-09a7e35718aa None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:41.488196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c48678-3987-4533-8f23-09a7e35718aa None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:41.489724 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c48678-3987-4533-8f23-09a7e35718aa None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:41.520511 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21d5c6e-a0dc-4c7d-97cd-38c377a0636f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:41.520776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 41/85] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:53:41 2022] POST /identity/v3/users => generated 289 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:53:41.521664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c48678-3987-4533-8f23-09a7e35718aa None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:41.573857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:41.594855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c988570d-49f9-4335-883f-4ca3e49a84a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:41.595917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c988570d-49f9-4335-883f-4ca3e49a84a5 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:41.596384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c988570d-49f9-4335-883f-4ca3e49a84a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:41.597428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c988570d-49f9-4335-883f-4ca3e49a84a5 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:41.598766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c988570d-49f9-4335-883f-4ca3e49a84a5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:41.599272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c988570d-49f9-4335-883f-4ca3e49a84a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:41.601859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c988570d-49f9-4335-883f-4ca3e49a84a5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:41.620695 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 45/86] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:53:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:41.620695 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c988570d-49f9-4335-883f-4ca3e49a84a5 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:41.631109 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:41.723161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c48678-3987-4533-8f23-09a7e35718aa None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:41.767638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd423b6-d3e5-451b-b3bf-21c31f98c6f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:41.769306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd423b6-d3e5-451b-b3bf-21c31f98c6f2 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:41.769821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd423b6-d3e5-451b-b3bf-21c31f98c6f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:41.770327 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd423b6-d3e5-451b-b3bf-21c31f98c6f2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:41.788625 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c988570d-49f9-4335-883f-4ca3e49a84a5 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:41.851215 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fb4bd5-5966-48f4-87f3-cd24c3d3a1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:41.851986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fb4bd5-5966-48f4-87f3-cd24c3d3a1c7 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:41.852271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fb4bd5-5966-48f4-87f3-cd24c3d3a1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:41.852549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fb4bd5-5966-48f4-87f3-cd24c3d3a1c7 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:41.853884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fb4bd5-5966-48f4-87f3-cd24c3d3a1c7 None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:41.855427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fb4bd5-5966-48f4-87f3-cd24c3d3a1c7 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:41.857488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 46/87] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:41 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:41.857928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fb4bd5-5966-48f4-87f3-cd24c3d3a1c7 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:41.895506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd423b6-d3e5-451b-b3bf-21c31f98c6f2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:41.898252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd423b6-d3e5-451b-b3bf-21c31f98c6f2 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:41.958554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:41.996657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:42.038657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fb4bd5-5966-48f4-87f3-cd24c3d3a1c7 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:42.106648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 42/88] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 585 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:53:42.108479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd423b6-d3e5-451b-b3bf-21c31f98c6f2 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:42.118804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6e58c9-8e1a-444a-bc73-31652da2d9dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:42.120019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e58c9-8e1a-444a-bc73-31652da2d9dd None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:42.120655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e58c9-8e1a-444a-bc73-31652da2d9dd None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:42.120943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e58c9-8e1a-444a-bc73-31652da2d9dd None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:42.130034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6e58c9-8e1a-444a-bc73-31652da2d9dd None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:42.132682 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6e58c9-8e1a-444a-bc73-31652da2d9dd None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:42.136509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 47/89] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:41 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:42.138347 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6e58c9-8e1a-444a-bc73-31652da2d9dd None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:42.348510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd423b6-d3e5-451b-b3bf-21c31f98c6f2 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:42.364975 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6e58c9-8e1a-444a-bc73-31652da2d9dd None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:42.401263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:42.417175 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92b64ec-88bd-45f3-b3f4-baf81afc1605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:42.418500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92b64ec-88bd-45f3-b3f4-baf81afc1605 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:42.419024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92b64ec-88bd-45f3-b3f4-baf81afc1605 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:42.419686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92b64ec-88bd-45f3-b3f4-baf81afc1605 None admin] PATH_INFO: `/v3/users/a614cb9f929748ecb70ea9ce0aed9916` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:42.421797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92b64ec-88bd-45f3-b3f4-baf81afc1605 None admin] RBAC: Authorizing `identity:get_user(user_id=a614cb9f929748ecb70ea9ce0aed9916)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:42.424578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92b64ec-88bd-45f3-b3f4-baf81afc1605 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:42.428848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 43/90] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:42 2022] GET /identity/v3/users/a614cb9f929748ecb70ea9ce0aed9916 => generated 275 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:42.437191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92b64ec-88bd-45f3-b3f4-baf81afc1605 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:42.446073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7776829b-6b89-4acf-87e5-6fcae9540363 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:42.447171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7776829b-6b89-4acf-87e5-6fcae9540363 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:42.447717 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7776829b-6b89-4acf-87e5-6fcae9540363 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:42.448204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7776829b-6b89-4acf-87e5-6fcae9540363 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:42.449688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7776829b-6b89-4acf-87e5-6fcae9540363 None admin] RBAC: Adding query filter params (role.id=ac68e1820c604122981fd5c4e2ebc447, scope.project.id=c0935b3a5a454c70bf75163763e11b34, user.id=a614cb9f929748ecb70ea9ce0aed9916) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:42.450259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7776829b-6b89-4acf-87e5-6fcae9540363 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:42.453065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7776829b-6b89-4acf-87e5-6fcae9540363 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:42.464849 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 48/91] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:53:42 2022] GET /identity/v3/role_assignments?user.id=a614cb9f929748ecb70ea9ce0aed9916&scope.project.id=c0935b3a5a454c70bf75163763e11b34&role.id=ac68e1820c604122981fd5c4e2ebc447 => generated 259 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:42.469793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7776829b-6b89-4acf-87e5-6fcae9540363 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:42.515804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:42.745939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92b64ec-88bd-45f3-b3f4-baf81afc1605 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:42.762403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7776829b-6b89-4acf-87e5-6fcae9540363 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:42.818115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80eb573-30ec-4846-9ba6-13c85e610fdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:42.820000 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80eb573-30ec-4846-9ba6-13c85e610fdf None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:42.820627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80eb573-30ec-4846-9ba6-13c85e610fdf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:42.821624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80eb573-30ec-4846-9ba6-13c85e610fdf None admin] PATH_INFO: `/v3/projects/9702636a96e94d8c8dbfe37dd9170280` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:42.828867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80eb573-30ec-4846-9ba6-13c85e610fdf None admin] RBAC: Authorizing `identity:get_project(project_id=9702636a96e94d8c8dbfe37dd9170280)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:42.833561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80eb573-30ec-4846-9ba6-13c85e610fdf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:42.843035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 44/92] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:42 2022] GET /identity/v3/projects/9702636a96e94d8c8dbfe37dd9170280 => generated 312 bytes in 406 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:42.843757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80eb573-30ec-4846-9ba6-13c85e610fdf None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:42.860943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53698455-7bce-480f-9cc2-b55407d73c96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:42.862037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53698455-7bce-480f-9cc2-b55407d73c96 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:42.863122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53698455-7bce-480f-9cc2-b55407d73c96 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:42.864848 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53698455-7bce-480f-9cc2-b55407d73c96 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:42.866421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53698455-7bce-480f-9cc2-b55407d73c96 None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=c0935b3a5a454c70bf75163763e11b34, user.id=e711c7fc76434a0b86b92dc8bcf69e87) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:42.867004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53698455-7bce-480f-9cc2-b55407d73c96 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:42.875504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53698455-7bce-480f-9cc2-b55407d73c96 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:42.900382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 49/93] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:53:42 2022] GET /identity/v3/role_assignments?user.id=e711c7fc76434a0b86b92dc8bcf69e87&scope.project.id=c0935b3a5a454c70bf75163763e11b34&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 259 bytes in 432 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:42.902140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53698455-7bce-480f-9cc2-b55407d73c96 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:42.992816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:43.042056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80eb573-30ec-4846-9ba6-13c85e610fdf None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:43.088051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9343480b-a090-4200-b31d-449b5e25dc96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:43.088942 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9343480b-a090-4200-b31d-449b5e25dc96 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:43.089225 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9343480b-a090-4200-b31d-449b5e25dc96 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:43.089503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9343480b-a090-4200-b31d-449b5e25dc96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:43.110262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:43.178062 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53698455-7bce-480f-9cc2-b55407d73c96 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:43.256018 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94820edc-b14c-419d-8581-89efd1cb69e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:43.256655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94820edc-b14c-419d-8581-89efd1cb69e5 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:43.256936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94820edc-b14c-419d-8581-89efd1cb69e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:43.257309 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94820edc-b14c-419d-8581-89efd1cb69e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:43.258216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94820edc-b14c-419d-8581-89efd1cb69e5 None admin] RBAC: Adding query filter params (role.id=ac68e1820c604122981fd5c4e2ebc447, scope.project.id=9702636a96e94d8c8dbfe37dd9170280, user.id=a614cb9f929748ecb70ea9ce0aed9916) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:43.258539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94820edc-b14c-419d-8581-89efd1cb69e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:43.260175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94820edc-b14c-419d-8581-89efd1cb69e5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:43.273813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9343480b-a090-4200-b31d-449b5e25dc96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:43.275518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9343480b-a090-4200-b31d-449b5e25dc96 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:43.279507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 50/94] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:53:42 2022] GET /identity/v3/role_assignments?user.id=a614cb9f929748ecb70ea9ce0aed9916&scope.project.id=9702636a96e94d8c8dbfe37dd9170280&role.id=ac68e1820c604122981fd5c4e2ebc447 => generated 259 bytes in 379 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:43.391982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:43.451522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 45/95] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 609 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:53:43.458617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94820edc-b14c-419d-8581-89efd1cb69e5 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:43.657118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94820edc-b14c-419d-8581-89efd1cb69e5 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:43.707475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a8411c-f776-47cd-bfd2-8cd597f52831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:43.708139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a8411c-f776-47cd-bfd2-8cd597f52831 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:43.708426 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a8411c-f776-47cd-bfd2-8cd597f52831 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:43.708713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a8411c-f776-47cd-bfd2-8cd597f52831 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:43.709956 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a8411c-f776-47cd-bfd2-8cd597f52831 None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:43.711464 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a8411c-f776-47cd-bfd2-8cd597f52831 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:43.713497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 51/96] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:43 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:43.719437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9343480b-a090-4200-b31d-449b5e25dc96 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:43.910067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9343480b-a090-4200-b31d-449b5e25dc96 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:43.971139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5387a0c6-289f-4fb2-8eec-d1b4a05fadc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:43.971860 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5387a0c6-289f-4fb2-8eec-d1b4a05fadc0 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:43.972161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5387a0c6-289f-4fb2-8eec-d1b4a05fadc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:43.972405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5387a0c6-289f-4fb2-8eec-d1b4a05fadc0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:43.973282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5387a0c6-289f-4fb2-8eec-d1b4a05fadc0 None admin] RBAC: Adding query filter params (role.id=0a2f5ca94533412088499eaccd7b2e21, scope.project.id=c0935b3a5a454c70bf75163763e11b34, user.id=a614cb9f929748ecb70ea9ce0aed9916) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:43.973595 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5387a0c6-289f-4fb2-8eec-d1b4a05fadc0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:43.975146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5387a0c6-289f-4fb2-8eec-d1b4a05fadc0 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:43.984208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 46/97] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:53:43 2022] GET /identity/v3/role_assignments?user.id=a614cb9f929748ecb70ea9ce0aed9916&scope.project.id=c0935b3a5a454c70bf75163763e11b34&role.id=0a2f5ca94533412088499eaccd7b2e21 => generated 259 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:44.335506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911d3a9b-1c17-4c88-9faa-ef3a1b01a197 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:44.335723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911d3a9b-1c17-4c88-9faa-ef3a1b01a197 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:44.335962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911d3a9b-1c17-4c88-9faa-ef3a1b01a197 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:44.337929 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 52/98] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:44.343669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d2b43b-c0d0-46d6-a6a4-267c82d531ca None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:44.343955 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d2b43b-c0d0-46d6-a6a4-267c82d531ca None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:44.344402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d2b43b-c0d0-46d6-a6a4-267c82d531ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:44.369967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-58d2b43b-c0d0-46d6-a6a4-267c82d531ca None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:44.371523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-58d2b43b-c0d0-46d6-a6a4-267c82d531ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:44.409461 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:44.504961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 47/99] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:44.558376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911d3a9b-1c17-4c88-9faa-ef3a1b01a197 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:44.568846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:44.712181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911d3a9b-1c17-4c88-9faa-ef3a1b01a197 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:44.765807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9d5a89-79c7-4c38-a66b-c19c7761c835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:44.766902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9d5a89-79c7-4c38-a66b-c19c7761c835 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:44.767397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9d5a89-79c7-4c38-a66b-c19c7761c835 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:44.767937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9d5a89-79c7-4c38-a66b-c19c7761c835 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:44.796077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9d5a89-79c7-4c38-a66b-c19c7761c835 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-9d9d5a89-79c7-4c38-a66b-c19c7761c835 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:44.803110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:44.804163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:44.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:44.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:44.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:44.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:44.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:44.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:44.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:44.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:44.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:44.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:44.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:53:44.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:53:44.807603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 53/100] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:44 2022] GET /identity/v3/roles/reader => generated 84 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:44.817084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d2b43b-c0d0-46d6-a6a4-267c82d531ca None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:44.967344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d2b43b-c0d0-46d6-a6a4-267c82d531ca None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:44.981981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f32bb3-60c2-4674-a167-593f2fe455bb None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:44.982297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f32bb3-60c2-4674-a167-593f2fe455bb None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:44.982631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f32bb3-60c2-4674-a167-593f2fe455bb None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:44.992239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 54/101] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:44 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:53:45.012973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35699808-4dbc-43ac-8741-26e4bb03ff8f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:45.013908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35699808-4dbc-43ac-8741-26e4bb03ff8f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:45.014658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35699808-4dbc-43ac-8741-26e4bb03ff8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:45.019201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dac168e-3a5b-4729-bdb9-38792a211ed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:45.020136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dac168e-3a5b-4729-bdb9-38792a211ed6 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:45.020555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dac168e-3a5b-4729-bdb9-38792a211ed6 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:45.020988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dac168e-3a5b-4729-bdb9-38792a211ed6 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:45.021852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dac168e-3a5b-4729-bdb9-38792a211ed6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:45.022282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dac168e-3a5b-4729-bdb9-38792a211ed6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:45.023842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dac168e-3a5b-4729-bdb9-38792a211ed6 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:45.033450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 48/102] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:53:44 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:45.041283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dac168e-3a5b-4729-bdb9-38792a211ed6 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:45.068495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-35699808-4dbc-43ac-8741-26e4bb03ff8f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:45.069705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-35699808-4dbc-43ac-8741-26e4bb03ff8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:45.072362 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:45.228006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dac168e-3a5b-4729-bdb9-38792a211ed6 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:45.255210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:45.275629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 55/103] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:45.304155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cab2200-1a9f-4289-890a-ec19e2a95e7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:45.305079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cab2200-1a9f-4289-890a-ec19e2a95e7b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:45.305508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cab2200-1a9f-4289-890a-ec19e2a95e7b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:45.305930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cab2200-1a9f-4289-890a-ec19e2a95e7b None admin] PATH_INFO: `/v3/users/acdd051bae9b454c99a875a522cdc828` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:45.332171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cab2200-1a9f-4289-890a-ec19e2a95e7b None admin] RBAC: Authorizing `identity:get_user(user_id=acdd051bae9b454c99a875a522cdc828)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:45.332643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35699808-4dbc-43ac-8741-26e4bb03ff8f None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:45.334107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cab2200-1a9f-4289-890a-ec19e2a95e7b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:45.336367 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 49/104] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:45 2022] GET /identity/v3/users/acdd051bae9b454c99a875a522cdc828 => generated 289 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:45.342774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cab2200-1a9f-4289-890a-ec19e2a95e7b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:45.456833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:45.488412 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cab2200-1a9f-4289-890a-ec19e2a95e7b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:45.589485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead3344a-0559-419d-ad05-ad3907919d1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:45.590278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead3344a-0559-419d-ad05-ad3907919d1a None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:45.590567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead3344a-0559-419d-ad05-ad3907919d1a None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:45.590843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead3344a-0559-419d-ad05-ad3907919d1a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:45.611731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35699808-4dbc-43ac-8741-26e4bb03ff8f None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:45.707949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bceb011d-a960-4d02-a77b-d09b940baed1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:45.708669 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bceb011d-a960-4d02-a77b-d09b940baed1 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:45.708970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bceb011d-a960-4d02-a77b-d09b940baed1 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:45.709287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bceb011d-a960-4d02-a77b-d09b940baed1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:45.710569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bceb011d-a960-4d02-a77b-d09b940baed1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:45.716246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bceb011d-a960-4d02-a77b-d09b940baed1 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:45.718137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 56/105] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:53:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 386 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:45.725180 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bceb011d-a960-4d02-a77b-d09b940baed1 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:45.759278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead3344a-0559-419d-ad05-ad3907919d1a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:45.761121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead3344a-0559-419d-ad05-ad3907919d1a None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:45.887772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:45.939898 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:45.963611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 50/106] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 621 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 31 00:53:45.969589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead3344a-0559-419d-ad05-ad3907919d1a None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:46.062583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bceb011d-a960-4d02-a77b-d09b940baed1 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:46.138562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e51297d-59ea-4094-a324-77fdcfd994f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:46.139458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e51297d-59ea-4094-a324-77fdcfd994f5 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:46.139749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e51297d-59ea-4094-a324-77fdcfd994f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:46.140158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e51297d-59ea-4094-a324-77fdcfd994f5 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:46.142645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e51297d-59ea-4094-a324-77fdcfd994f5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:46.144836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e51297d-59ea-4094-a324-77fdcfd994f5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:46.184519 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead3344a-0559-419d-ad05-ad3907919d1a None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:46.195889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 57/107] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:53:45 2022] POST /identity/v3/projects => generated 302 bytes in 471 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:53:46.202410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af5d3f2-f237-4e9e-b250-14e6b64b2c54 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:46.202746 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af5d3f2-f237-4e9e-b250-14e6b64b2c54 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:46.202998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af5d3f2-f237-4e9e-b250-14e6b64b2c54 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:46.204531 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 58/108] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:46 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:46.212513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ffcda8-97ea-44ac-8496-1c43fae2193d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:46.212843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ffcda8-97ea-44ac-8496-1c43fae2193d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:46.213090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ffcda8-97ea-44ac-8496-1c43fae2193d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:46.253501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d6ffcda8-97ea-44ac-8496-1c43fae2193d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:46.254773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d6ffcda8-97ea-44ac-8496-1c43fae2193d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:46.269597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499bdb60-d2de-41ce-97c6-2a5aeac74288 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:46.270398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499bdb60-d2de-41ce-97c6-2a5aeac74288 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:46.270687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499bdb60-d2de-41ce-97c6-2a5aeac74288 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:46.271011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499bdb60-d2de-41ce-97c6-2a5aeac74288 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:46.273680 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499bdb60-d2de-41ce-97c6-2a5aeac74288 None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:46.276506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499bdb60-d2de-41ce-97c6-2a5aeac74288 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:46.280463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 51/109] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:45 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:46.294176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499bdb60-d2de-41ce-97c6-2a5aeac74288 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:46.389217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:46.392486 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 59/110] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:53:46.436932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ffcda8-97ea-44ac-8496-1c43fae2193d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:46.493618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499bdb60-d2de-41ce-97c6-2a5aeac74288 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:46.515649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:46.557473 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9616a50c-91e6-4c1d-b9c1-9d4fbbef4434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:46.558220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9616a50c-91e6-4c1d-b9c1-9d4fbbef4434 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:46.558504 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9616a50c-91e6-4c1d-b9c1-9d4fbbef4434 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:46.558787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9616a50c-91e6-4c1d-b9c1-9d4fbbef4434 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:46.559673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9616a50c-91e6-4c1d-b9c1-9d4fbbef4434 None admin] RBAC: Adding query filter params (role.id=de6529a9de0b48b2910599a3a56124f3, scope.project.id=c0935b3a5a454c70bf75163763e11b34, user.id=acdd051bae9b454c99a875a522cdc828) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:46.559979 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9616a50c-91e6-4c1d-b9c1-9d4fbbef4434 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:46.561716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9616a50c-91e6-4c1d-b9c1-9d4fbbef4434 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:46.570219 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 52/111] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:53:46 2022] GET /identity/v3/role_assignments?user.id=acdd051bae9b454c99a875a522cdc828&scope.project.id=c0935b3a5a454c70bf75163763e11b34&role.id=de6529a9de0b48b2910599a3a56124f3 => generated 259 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:46.572020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbfada6-a3fd-430e-b3c6-7c021c5513bf None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:46.572625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbfada6-a3fd-430e-b3c6-7c021c5513bf None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:46.573109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbfada6-a3fd-430e-b3c6-7c021c5513bf None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:46.575228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 53/112] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:46.577297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73ddfb5-e134-452b-8c86-ddd843e891c6 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:46.577954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73ddfb5-e134-452b-8c86-ddd843e891c6 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:46.578535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73ddfb5-e134-452b-8c86-ddd843e891c6 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:46.580541 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 54/113] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:46.586571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9cf271-3f94-46ce-835a-9d41bb25e5cd None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:46.587503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9cf271-3f94-46ce-835a-9d41bb25e5cd None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:46.588304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9cf271-3f94-46ce-835a-9d41bb25e5cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:46.604755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ffcda8-97ea-44ac-8496-1c43fae2193d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:46.620916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-cc9cf271-3f94-46ce-835a-9d41bb25e5cd None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:46.623105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-cc9cf271-3f94-46ce-835a-9d41bb25e5cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:46.651774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79c6162-d409-4b3b-88da-0f48f5c85bbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:46.652649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c6162-d409-4b3b-88da-0f48f5c85bbd None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:46.653365 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c6162-d409-4b3b-88da-0f48f5c85bbd None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:46.653891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c6162-d409-4b3b-88da-0f48f5c85bbd None admin] PATH_INFO: `/v3/roles/member` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:46.670894 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79c6162-d409-4b3b-88da-0f48f5c85bbd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-e79c6162-d409-4b3b-88da-0f48f5c85bbd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:46.680812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:46.681980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:46.682941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:46.682941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:46.682941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:46.682941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:46.682941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:46.682941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:46.682941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:46.682941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:46.682941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:46.682941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:46.682941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:53:46.682941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:53:46.685649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 60/114] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:53:46.687297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b412d-e43c-4086-8877-6e6d7c380144 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:46.687786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b412d-e43c-4086-8877-6e6d7c380144 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:46.688259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b412d-e43c-4086-8877-6e6d7c380144 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:46.710314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6d9b412d-e43c-4086-8877-6e6d7c380144 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:46.711916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6d9b412d-e43c-4086-8877-6e6d7c380144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:46.741604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 55/115] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:46.742945 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9cf271-3f94-46ce-835a-9d41bb25e5cd None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:46.801820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 61/116] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:46.803024 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9b412d-e43c-4086-8877-6e6d7c380144 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:46.832697 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:46.846076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:46.940371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9b412d-e43c-4086-8877-6e6d7c380144 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:46.950858 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9cf271-3f94-46ce-835a-9d41bb25e5cd None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:46.988988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36bec3f-29a7-4da4-8fc1-c51d9bb697b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:46.989945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36bec3f-29a7-4da4-8fc1-c51d9bb697b5 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:46.990688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36bec3f-29a7-4da4-8fc1-c51d9bb697b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:46.991929 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36bec3f-29a7-4da4-8fc1-c51d9bb697b5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:47.014068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36bec3f-29a7-4da4-8fc1-c51d9bb697b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-f36bec3f-29a7-4da4-8fc1-c51d9bb697b5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:47.024676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:47.025736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:47.026788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:47.026788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:47.026788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:47.026788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:47.026788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:47.026788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:47.026788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:47.026788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:47.026788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:47.026788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:47.026788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:47.026788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:53:47.033136 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 62/117] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:53:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:47.035270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36bec3f-29a7-4da4-8fc1-c51d9bb697b5 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:47.056128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ac2d49-55f3-4ca1-b3cf-f06fe9b590f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:47.056763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ac2d49-55f3-4ca1-b3cf-f06fe9b590f5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:47.057049 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ac2d49-55f3-4ca1-b3cf-f06fe9b590f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:47.057552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ac2d49-55f3-4ca1-b3cf-f06fe9b590f5 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:47.058060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ac2d49-55f3-4ca1-b3cf-f06fe9b590f5 None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:47.058372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ac2d49-55f3-4ca1-b3cf-f06fe9b590f5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:47.059843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ac2d49-55f3-4ca1-b3cf-f06fe9b590f5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:47.079904 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 56/118] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:53:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 338 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:47.080902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ac2d49-55f3-4ca1-b3cf-f06fe9b590f5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:47.150378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:47.156620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36bec3f-29a7-4da4-8fc1-c51d9bb697b5 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:47.257895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1724ebb-1036-48ba-9f3b-8973237ef092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:47.260271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1724ebb-1036-48ba-9f3b-8973237ef092 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:47.260715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1724ebb-1036-48ba-9f3b-8973237ef092 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:47.261193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1724ebb-1036-48ba-9f3b-8973237ef092 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:47.269649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:47.275577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ac2d49-55f3-4ca1-b3cf-f06fe9b590f5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:47.290386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1724ebb-1036-48ba-9f3b-8973237ef092 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-d1724ebb-1036-48ba-9f3b-8973237ef092 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:47.296457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:47.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:53:47.298488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 63/119] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:47.299275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1724ebb-1036-48ba-9f3b-8973237ef092 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:47.333503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e425e8f-6c43-4fad-b2e9-be37a056a528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:47.334817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e425e8f-6c43-4fad-b2e9-be37a056a528 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:47.335421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e425e8f-6c43-4fad-b2e9-be37a056a528 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:47.335893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e425e8f-6c43-4fad-b2e9-be37a056a528 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:47.336990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e425e8f-6c43-4fad-b2e9-be37a056a528 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:47.337479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e425e8f-6c43-4fad-b2e9-be37a056a528 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:47.339634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e425e8f-6c43-4fad-b2e9-be37a056a528 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:47.351418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 57/120] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:47.353055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e425e8f-6c43-4fad-b2e9-be37a056a528 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:47.427944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1724ebb-1036-48ba-9f3b-8973237ef092 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:47.466679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38a49b7-7940-422a-9c9f-8e2581c04857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:47.467328 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38a49b7-7940-422a-9c9f-8e2581c04857 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:47.467641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38a49b7-7940-422a-9c9f-8e2581c04857 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:47.467921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38a49b7-7940-422a-9c9f-8e2581c04857 None admin] PATH_INFO: `/v3/users/a614cb9f929748ecb70ea9ce0aed9916` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:47.469135 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38a49b7-7940-422a-9c9f-8e2581c04857 None admin] RBAC: Authorizing `identity:get_user(user_id=a614cb9f929748ecb70ea9ce0aed9916)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:47.470667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38a49b7-7940-422a-9c9f-8e2581c04857 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:47.479302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 64/121] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:47 2022] GET /identity/v3/users/a614cb9f929748ecb70ea9ce0aed9916 => generated 275 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:47.479302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38a49b7-7940-422a-9c9f-8e2581c04857 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:47.503560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:47.525887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e425e8f-6c43-4fad-b2e9-be37a056a528 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:47.585861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5b4d36-500a-484a-8a2f-3cb5aae75c94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:47.586785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5b4d36-500a-484a-8a2f-3cb5aae75c94 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:47.587239 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5b4d36-500a-484a-8a2f-3cb5aae75c94 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:47.587693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5b4d36-500a-484a-8a2f-3cb5aae75c94 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:47.589354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5b4d36-500a-484a-8a2f-3cb5aae75c94 None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:47.589833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5b4d36-500a-484a-8a2f-3cb5aae75c94 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:47.591394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5b4d36-500a-484a-8a2f-3cb5aae75c94 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:47.602480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 58/122] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:53:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:47.609022 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128e5b12-c9f0-4628-a178-94df505e2c10 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:47.609623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128e5b12-c9f0-4628-a178-94df505e2c10 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:47.610082 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128e5b12-c9f0-4628-a178-94df505e2c10 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:47.612702 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 59/123] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:47.618939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128e5b12-c9f0-4628-a178-94df505e2c10 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:47.633513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38a49b7-7940-422a-9c9f-8e2581c04857 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:47.657080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:47.672524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd6e66b-5935-4bfb-a2a6-a9e9fe6e09a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:47.673871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd6e66b-5935-4bfb-a2a6-a9e9fe6e09a2 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:47.674333 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd6e66b-5935-4bfb-a2a6-a9e9fe6e09a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:47.675419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd6e66b-5935-4bfb-a2a6-a9e9fe6e09a2 None admin] PATH_INFO: `/v3/users/e711c7fc76434a0b86b92dc8bcf69e87` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:47.677547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd6e66b-5935-4bfb-a2a6-a9e9fe6e09a2 None admin] RBAC: Authorizing `identity:get_user(user_id=e711c7fc76434a0b86b92dc8bcf69e87)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:47.679609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd6e66b-5935-4bfb-a2a6-a9e9fe6e09a2 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:47.681705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 65/124] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:47 2022] GET /identity/v3/users/e711c7fc76434a0b86b92dc8bcf69e87 => generated 247 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:47.682695 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd6e66b-5935-4bfb-a2a6-a9e9fe6e09a2 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:47.753730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128e5b12-c9f0-4628-a178-94df505e2c10 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:47.787491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:47.793861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c02c51e-44c9-455b-b64c-6033c349004a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:47.794637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c02c51e-44c9-455b-b64c-6033c349004a None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:47.794997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c02c51e-44c9-455b-b64c-6033c349004a None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:47.795388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c02c51e-44c9-455b-b64c-6033c349004a None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:47.796761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c02c51e-44c9-455b-b64c-6033c349004a None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:47.798492 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c02c51e-44c9-455b-b64c-6033c349004a None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:47.801056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 60/125] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:47 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:47.802840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd6e66b-5935-4bfb-a2a6-a9e9fe6e09a2 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:47.807283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54947f6d-fe5f-42d2-aa1f-0b4bf5b52d0b None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:47.807673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54947f6d-fe5f-42d2-aa1f-0b4bf5b52d0b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:47.808030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54947f6d-fe5f-42d2-aa1f-0b4bf5b52d0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:47.829550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-54947f6d-fe5f-42d2-aa1f-0b4bf5b52d0b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:47.830918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-54947f6d-fe5f-42d2-aa1f-0b4bf5b52d0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:47.850362 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada54515-adaf-4d24-b775-c9ed55835474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:47.851321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada54515-adaf-4d24-b775-c9ed55835474 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:47.851812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada54515-adaf-4d24-b775-c9ed55835474 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:47.852259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada54515-adaf-4d24-b775-c9ed55835474 None admin] PATH_INFO: `/v3/users/a614cb9f929748ecb70ea9ce0aed9916` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:47.853693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada54515-adaf-4d24-b775-c9ed55835474 None admin] RBAC: Authorizing `identity:get_user(user_id=a614cb9f929748ecb70ea9ce0aed9916)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:47.855389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada54515-adaf-4d24-b775-c9ed55835474 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:47.857264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 66/126] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:47 2022] GET /identity/v3/users/a614cb9f929748ecb70ea9ce0aed9916 => generated 275 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:47.863009 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada54515-adaf-4d24-b775-c9ed55835474 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:47.910961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 61/127] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:47.913042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54947f6d-fe5f-42d2-aa1f-0b4bf5b52d0b None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:47.978656 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:48.022703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada54515-adaf-4d24-b775-c9ed55835474 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:48.069445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a5919a-44ba-4482-998d-ddf6785074cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:48.070767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a5919a-44ba-4482-998d-ddf6785074cc None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:48.071292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a5919a-44ba-4482-998d-ddf6785074cc None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:48.071862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a5919a-44ba-4482-998d-ddf6785074cc None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:48.073526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a5919a-44ba-4482-998d-ddf6785074cc None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:48.075341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a5919a-44ba-4482-998d-ddf6785074cc None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:48.076039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54947f6d-fe5f-42d2-aa1f-0b4bf5b52d0b None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:48.078302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 67/128] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:47 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:48.079611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a5919a-44ba-4482-998d-ddf6785074cc None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:48.115636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:48.120208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d97fbf-58bb-45d3-9da9-3597e4a9a952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:48.120935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d97fbf-58bb-45d3-9da9-3597e4a9a952 None admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:48.121223 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d97fbf-58bb-45d3-9da9-3597e4a9a952 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:48.122097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d97fbf-58bb-45d3-9da9-3597e4a9a952 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34/users/a614cb9f929748ecb70ea9ce0aed9916/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:48.124242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d97fbf-58bb-45d3-9da9-3597e4a9a952 None admin] RBAC: Authorizing `identity:create_grant(project_id=c0935b3a5a454c70bf75163763e11b34, user_id=a614cb9f929748ecb70ea9ce0aed9916, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:48.125864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d97fbf-58bb-45d3-9da9-3597e4a9a952 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:48.140758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 62/129] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:53:47 2022] PUT /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34/users/a614cb9f929748ecb70ea9ce0aed9916/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:53:48.141630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d97fbf-58bb-45d3-9da9-3597e4a9a952 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:48.213577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a5919a-44ba-4482-998d-ddf6785074cc None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:48.248041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:48.275423 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f0f20e-1d0f-48ab-ad71-e4b685e0bc4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:48.276699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f0f20e-1d0f-48ab-ad71-e4b685e0bc4c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:48.277389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f0f20e-1d0f-48ab-ad71-e4b685e0bc4c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:48.277972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f0f20e-1d0f-48ab-ad71-e4b685e0bc4c None admin] PATH_INFO: `/v3/projects/9702636a96e94d8c8dbfe37dd9170280` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:48.279984 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f0f20e-1d0f-48ab-ad71-e4b685e0bc4c None admin] RBAC: Authorizing `identity:get_project(project_id=9702636a96e94d8c8dbfe37dd9170280)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:48.282745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f0f20e-1d0f-48ab-ad71-e4b685e0bc4c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:48.286381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 68/130] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:48 2022] GET /identity/v3/projects/9702636a96e94d8c8dbfe37dd9170280 => generated 312 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:48.292718 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f0f20e-1d0f-48ab-ad71-e4b685e0bc4c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:48.292953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d97fbf-58bb-45d3-9da9-3597e4a9a952 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:48.365591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8828d09a-2f05-441d-9ee6-2ef0d4f5b32f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:48.366270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8828d09a-2f05-441d-9ee6-2ef0d4f5b32f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:48.366560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8828d09a-2f05-441d-9ee6-2ef0d4f5b32f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:48.366835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8828d09a-2f05-441d-9ee6-2ef0d4f5b32f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:48.388251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8828d09a-2f05-441d-9ee6-2ef0d4f5b32f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-8828d09a-2f05-441d-9ee6-2ef0d4f5b32f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:48.395889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:48.396896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:53:48.397809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 63/131] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:53:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:48.398586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8828d09a-2f05-441d-9ee6-2ef0d4f5b32f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:48.505159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f0f20e-1d0f-48ab-ad71-e4b685e0bc4c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:48.518869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:48.571883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08e162d-9756-4f9c-b59b-9d2f7833fb5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:48.572730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08e162d-9756-4f9c-b59b-9d2f7833fb5a None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:48.573257 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08e162d-9756-4f9c-b59b-9d2f7833fb5a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:48.573779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08e162d-9756-4f9c-b59b-9d2f7833fb5a None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34/users/e711c7fc76434a0b86b92dc8bcf69e87/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:48.579276 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08e162d-9756-4f9c-b59b-9d2f7833fb5a None admin] RBAC: Authorizing `identity:create_grant(project_id=c0935b3a5a454c70bf75163763e11b34, user_id=e711c7fc76434a0b86b92dc8bcf69e87, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:48.582263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08e162d-9756-4f9c-b59b-9d2f7833fb5a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:48.594721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8828d09a-2f05-441d-9ee6-2ef0d4f5b32f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:48.626137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 69/132] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:53:48 2022] PUT /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34/users/e711c7fc76434a0b86b92dc8bcf69e87/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:53:48.626908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08e162d-9756-4f9c-b59b-9d2f7833fb5a None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:48.666299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbdaea6-60cd-4857-9fb4-5ac54d326eb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:48.667805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbdaea6-60cd-4857-9fb4-5ac54d326eb5 None admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:48.668319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbdaea6-60cd-4857-9fb4-5ac54d326eb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:48.668808 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbdaea6-60cd-4857-9fb4-5ac54d326eb5 None admin] PATH_INFO: `/v3/projects/9702636a96e94d8c8dbfe37dd9170280/users/a614cb9f929748ecb70ea9ce0aed9916/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:48.672537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bbdaea6-60cd-4857-9fb4-5ac54d326eb5 None admin] RBAC: Authorizing `identity:create_grant(project_id=9702636a96e94d8c8dbfe37dd9170280, user_id=a614cb9f929748ecb70ea9ce0aed9916, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:48.676177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bbdaea6-60cd-4857-9fb4-5ac54d326eb5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:48.709106 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 64/133] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:53:48 2022] PUT /identity/v3/projects/9702636a96e94d8c8dbfe37dd9170280/users/a614cb9f929748ecb70ea9ce0aed9916/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:53:48.840254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08e162d-9756-4f9c-b59b-9d2f7833fb5a None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:48.889951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8734001a-c8c8-40e3-beaa-1d96b387f879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:48.890530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8734001a-c8c8-40e3-beaa-1d96b387f879 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:48.890819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8734001a-c8c8-40e3-beaa-1d96b387f879 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:48.891099 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8734001a-c8c8-40e3-beaa-1d96b387f879 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:48.891916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8734001a-c8c8-40e3-beaa-1d96b387f879 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:48.892229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8734001a-c8c8-40e3-beaa-1d96b387f879 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:48.893863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8734001a-c8c8-40e3-beaa-1d96b387f879 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:48.905715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 70/134] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:53:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:48.912476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbdaea6-60cd-4857-9fb4-5ac54d326eb5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:48.921714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:48.942321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18555cf-fc50-469b-8342-c101906531f7 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:48.942639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18555cf-fc50-469b-8342-c101906531f7 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:48.942892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18555cf-fc50-469b-8342-c101906531f7 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:48.944372 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 71/135] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:53:48.950587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea52337-16d0-4f04-9f4b-b5a315dbaf9d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:48.950927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea52337-16d0-4f04-9f4b-b5a315dbaf9d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:48.951151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea52337-16d0-4f04-9f4b-b5a315dbaf9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:48.971557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-dea52337-16d0-4f04-9f4b-b5a315dbaf9d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:48.973330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-dea52337-16d0-4f04-9f4b-b5a315dbaf9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:48.997060 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:49.075552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 72/136] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:53:49.109259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea52337-16d0-4f04-9f4b-b5a315dbaf9d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:49.130853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbdaea6-60cd-4857-9fb4-5ac54d326eb5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:49.214930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7741a2ae-aa9f-4826-98ac-da4ec24f070c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:49.215634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7741a2ae-aa9f-4826-98ac-da4ec24f070c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:49.215925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7741a2ae-aa9f-4826-98ac-da4ec24f070c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:49.216205 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7741a2ae-aa9f-4826-98ac-da4ec24f070c None admin] PATH_INFO: `/v3/users/a614cb9f929748ecb70ea9ce0aed9916` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:49.217439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7741a2ae-aa9f-4826-98ac-da4ec24f070c None admin] RBAC: Authorizing `identity:get_user(user_id=a614cb9f929748ecb70ea9ce0aed9916)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:49.218906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7741a2ae-aa9f-4826-98ac-da4ec24f070c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:49.226103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 65/137] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:48 2022] GET /identity/v3/users/a614cb9f929748ecb70ea9ce0aed9916 => generated 275 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:49.229497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7741a2ae-aa9f-4826-98ac-da4ec24f070c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:49.313765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea52337-16d0-4f04-9f4b-b5a315dbaf9d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:49.382004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742564f5-de32-4adb-9bf4-656057428b05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:49.383058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742564f5-de32-4adb-9bf4-656057428b05 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:49.384210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742564f5-de32-4adb-9bf4-656057428b05 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:49.384726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742564f5-de32-4adb-9bf4-656057428b05 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:49.386210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742564f5-de32-4adb-9bf4-656057428b05 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:49.387954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742564f5-de32-4adb-9bf4-656057428b05 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:49.396736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 73/138] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:53:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:49.403556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742564f5-de32-4adb-9bf4-656057428b05 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:49.433357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:49.442201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:49.490086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7741a2ae-aa9f-4826-98ac-da4ec24f070c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:49.568894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7c3180-22eb-45cb-b5e3-40908d49acc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:49.570210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c3180-22eb-45cb-b5e3-40908d49acc7 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:49.570734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c3180-22eb-45cb-b5e3-40908d49acc7 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:49.571332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c3180-22eb-45cb-b5e3-40908d49acc7 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:49.573260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c3180-22eb-45cb-b5e3-40908d49acc7 None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:49.575424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c3180-22eb-45cb-b5e3-40908d49acc7 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:49.578011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 66/139] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:49 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 349 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:49.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2fe79e-d92a-43c8-b179-e21adb5e166a None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:49.580689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2fe79e-d92a-43c8-b179-e21adb5e166a None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:49.581867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2fe79e-d92a-43c8-b179-e21adb5e166a None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:49.584382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 67/140] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:49.592039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2abe85-0868-4943-b161-c7158e34c075 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:49.592732 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2abe85-0868-4943-b161-c7158e34c075 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:49.593333 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2abe85-0868-4943-b161-c7158e34c075 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:49.615524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-eb2abe85-0868-4943-b161-c7158e34c075 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:49.617180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-eb2abe85-0868-4943-b161-c7158e34c075 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:49.631725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742564f5-de32-4adb-9bf4-656057428b05 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:49.688068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323eb7c4-98a6-4ab7-a716-c8d616c0a9a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:49.689309 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323eb7c4-98a6-4ab7-a716-c8d616c0a9a5 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:49.689815 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323eb7c4-98a6-4ab7-a716-c8d616c0a9a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:49.690936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323eb7c4-98a6-4ab7-a716-c8d616c0a9a5 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:49.692314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323eb7c4-98a6-4ab7-a716-c8d616c0a9a5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:49.694222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323eb7c4-98a6-4ab7-a716-c8d616c0a9a5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:49.731814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 74/141] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:53:49 2022] POST /identity/v3/users => generated 283 bytes in 329 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 00:53:49.733356 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323eb7c4-98a6-4ab7-a716-c8d616c0a9a5 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:49.837514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 68/142] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:53:49.931813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:49.968462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2abe85-0868-4943-b161-c7158e34c075 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:49.969233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323eb7c4-98a6-4ab7-a716-c8d616c0a9a5 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:50.009863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad42842f-a081-4971-8da3-3a182bb3410f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:50.010635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad42842f-a081-4971-8da3-3a182bb3410f None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:50.010914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad42842f-a081-4971-8da3-3a182bb3410f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:50.011189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad42842f-a081-4971-8da3-3a182bb3410f None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34/users/a614cb9f929748ecb70ea9ce0aed9916/roles/0a2f5ca94533412088499eaccd7b2e21` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:50.013592 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad42842f-a081-4971-8da3-3a182bb3410f None admin] RBAC: Authorizing `identity:create_grant(project_id=c0935b3a5a454c70bf75163763e11b34, user_id=a614cb9f929748ecb70ea9ce0aed9916, role_id=0a2f5ca94533412088499eaccd7b2e21)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:50.015076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad42842f-a081-4971-8da3-3a182bb3410f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:50.038556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 75/143] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:53:49 2022] PUT /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34/users/a614cb9f929748ecb70ea9ce0aed9916/roles/0a2f5ca94533412088499eaccd7b2e21 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:53:50.087801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:50.142052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2abe85-0868-4943-b161-c7158e34c075 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:50.195590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c14d9e-3006-4563-9081-8106f8ec37a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:50.196306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c14d9e-3006-4563-9081-8106f8ec37a0 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:50.196586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c14d9e-3006-4563-9081-8106f8ec37a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:50.196857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c14d9e-3006-4563-9081-8106f8ec37a0 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:50.209560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c14d9e-3006-4563-9081-8106f8ec37a0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-22c14d9e-3006-4563-9081-8106f8ec37a0 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:50.215151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:50.216145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:53:50.216914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 69/144] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:49 2022] GET /identity/v3/roles/reader => generated 84 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:50.224108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad42842f-a081-4971-8da3-3a182bb3410f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:50.365326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad42842f-a081-4971-8da3-3a182bb3410f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:50.408044 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:50.409698 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe54756-c54c-4bf8-8556-0f32f8c95979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:50.410212 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe54756-c54c-4bf8-8556-0f32f8c95979 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:50.410814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe54756-c54c-4bf8-8556-0f32f8c95979 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:50.411082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe54756-c54c-4bf8-8556-0f32f8c95979 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:50.412105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe54756-c54c-4bf8-8556-0f32f8c95979 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:50.412429 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe54756-c54c-4bf8-8556-0f32f8c95979 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:50.414562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe54756-c54c-4bf8-8556-0f32f8c95979 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:50.427384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 76/145] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:53:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:50.434297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c14d9e-3006-4563-9081-8106f8ec37a0 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:50.642375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c14d9e-3006-4563-9081-8106f8ec37a0 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:50.678373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:50.694687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832e8351-3ef7-4c1f-b151-f18bc9f0af99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:50.695380 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832e8351-3ef7-4c1f-b151-f18bc9f0af99 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:50.695675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832e8351-3ef7-4c1f-b151-f18bc9f0af99 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:50.696030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832e8351-3ef7-4c1f-b151-f18bc9f0af99 None admin] PATH_INFO: `/v3/users/acdd051bae9b454c99a875a522cdc828` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:50.697211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832e8351-3ef7-4c1f-b151-f18bc9f0af99 None admin] RBAC: Authorizing `identity:get_user(user_id=acdd051bae9b454c99a875a522cdc828)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:50.698703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832e8351-3ef7-4c1f-b151-f18bc9f0af99 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:50.700427 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 70/146] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:50 2022] GET /identity/v3/users/acdd051bae9b454c99a875a522cdc828 => generated 289 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:50.711181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe54756-c54c-4bf8-8556-0f32f8c95979 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:50.909829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe54756-c54c-4bf8-8556-0f32f8c95979 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:50.978419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8241f7a7-6ab6-498d-9ffa-659b1673175d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:50.979592 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8241f7a7-6ab6-498d-9ffa-659b1673175d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:50.980246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8241f7a7-6ab6-498d-9ffa-659b1673175d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:50.981204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8241f7a7-6ab6-498d-9ffa-659b1673175d None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:50.981936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f35251-847b-4412-af5b-80b70e792d1b None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:50.982312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f35251-847b-4412-af5b-80b70e792d1b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:50.982574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f35251-847b-4412-af5b-80b70e792d1b None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:50.983819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8241f7a7-6ab6-498d-9ffa-659b1673175d None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:50.983973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 71/147] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:50.986007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8241f7a7-6ab6-498d-9ffa-659b1673175d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:50.988142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 77/148] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:50 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:50.997148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7da40e8-3040-4eda-9835-9adcc1a0d7cd None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:50.997462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7da40e8-3040-4eda-9835-9adcc1a0d7cd None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:50.997717 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7da40e8-3040-4eda-9835-9adcc1a0d7cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:50.998347 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8241f7a7-6ab6-498d-9ffa-659b1673175d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:51.044581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a7da40e8-3040-4eda-9835-9adcc1a0d7cd None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:51.045891 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a7da40e8-3040-4eda-9835-9adcc1a0d7cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:51.117062 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:51.197013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 72/149] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:51.201285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8241f7a7-6ab6-498d-9ffa-659b1673175d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:51.256445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b75634c-9ddb-4df8-867d-01dc1e02881f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:51.257570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b75634c-9ddb-4df8-867d-01dc1e02881f None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:51.258107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b75634c-9ddb-4df8-867d-01dc1e02881f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:51.258645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b75634c-9ddb-4df8-867d-01dc1e02881f None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34/users/acdd051bae9b454c99a875a522cdc828/roles/de6529a9de0b48b2910599a3a56124f3` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:51.262877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b75634c-9ddb-4df8-867d-01dc1e02881f None admin] RBAC: Authorizing `identity:create_grant(project_id=c0935b3a5a454c70bf75163763e11b34, user_id=acdd051bae9b454c99a875a522cdc828, role_id=de6529a9de0b48b2910599a3a56124f3)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:51.264854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b75634c-9ddb-4df8-867d-01dc1e02881f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:51.278378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7da40e8-3040-4eda-9835-9adcc1a0d7cd None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:51.291186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 78/150] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:53:50 2022] PUT /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34/users/acdd051bae9b454c99a875a522cdc828/roles/de6529a9de0b48b2910599a3a56124f3 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 00:53:51.411804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7da40e8-3040-4eda-9835-9adcc1a0d7cd None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:51.450794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:51.464165 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1780da44-66b8-4d9c-ba7c-e8b72a73dee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:51.464849 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780da44-66b8-4d9c-ba7c-e8b72a73dee3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:51.465137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780da44-66b8-4d9c-ba7c-e8b72a73dee3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:51.465470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780da44-66b8-4d9c-ba7c-e8b72a73dee3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:51.481664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1780da44-66b8-4d9c-ba7c-e8b72a73dee3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-1780da44-66b8-4d9c-ba7c-e8b72a73dee3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:51.487464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:51.488376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:53:51.490441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 73/151] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:53:51.495626 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b75634c-9ddb-4df8-867d-01dc1e02881f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:51.655535 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b75634c-9ddb-4df8-867d-01dc1e02881f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:51.684502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:51.702608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be66f52d-80fc-454f-99f8-0f14b9746a84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:51.703177 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be66f52d-80fc-454f-99f8-0f14b9746a84 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:51.703557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be66f52d-80fc-454f-99f8-0f14b9746a84 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:51.703763 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be66f52d-80fc-454f-99f8-0f14b9746a84 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:51.704547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be66f52d-80fc-454f-99f8-0f14b9746a84 None admin] RBAC: Adding query filter params (name=member) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:51.704855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be66f52d-80fc-454f-99f8-0f14b9746a84 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:51.706472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be66f52d-80fc-454f-99f8-0f14b9746a84 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:51.717714 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 79/152] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:53:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:51.724222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1780da44-66b8-4d9c-ba7c-e8b72a73dee3 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:51.856244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1780da44-66b8-4d9c-ba7c-e8b72a73dee3 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:51.905705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f86620-452f-4736-86ae-4743b7507668 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:51.906373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f86620-452f-4736-86ae-4743b7507668 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:51.906651 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f86620-452f-4736-86ae-4743b7507668 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:51.906926 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f86620-452f-4736-86ae-4743b7507668 None admin] PATH_INFO: `/v3/users/a614cb9f929748ecb70ea9ce0aed9916` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:51.908196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f86620-452f-4736-86ae-4743b7507668 None admin] RBAC: Authorizing `identity:get_user(user_id=a614cb9f929748ecb70ea9ce0aed9916)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:51.910907 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f86620-452f-4736-86ae-4743b7507668 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:51.913530 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 74/153] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:51 2022] GET /identity/v3/users/a614cb9f929748ecb70ea9ce0aed9916 => generated 275 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:51.919703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be66f52d-80fc-454f-99f8-0f14b9746a84 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:52.086791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be66f52d-80fc-454f-99f8-0f14b9746a84 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:52.138520 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7ea7fe-fd02-4555-81b6-961db1ef2069 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:52.139186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7ea7fe-fd02-4555-81b6-961db1ef2069 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:52.139566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7ea7fe-fd02-4555-81b6-961db1ef2069 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:52.139968 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7ea7fe-fd02-4555-81b6-961db1ef2069 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:52.218268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:52.239567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7ea7fe-fd02-4555-81b6-961db1ef2069 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:52.241023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7ea7fe-fd02-4555-81b6-961db1ef2069 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:52.393851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 80/154] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 474 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:53:52.401391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f86620-452f-4736-86ae-4743b7507668 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:52.471752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:52.551875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb73d31-e383-4126-a97e-ca932c52c4f7 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:52.552229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb73d31-e383-4126-a97e-ca932c52c4f7 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:52.552603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb73d31-e383-4126-a97e-ca932c52c4f7 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:52.554320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 81/155] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:52.555684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f86620-452f-4736-86ae-4743b7507668 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:52.561570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb1f7bb-bbd1-47d2-b0cf-1355fbd34383 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:52.561859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb1f7bb-bbd1-47d2-b0cf-1355fbd34383 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:52.562125 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb1f7bb-bbd1-47d2-b0cf-1355fbd34383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:52.594323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-feb1f7bb-bbd1-47d2-b0cf-1355fbd34383 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:52.595583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-feb1f7bb-bbd1-47d2-b0cf-1355fbd34383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:52.652970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da5ec9b-1d0e-4d79-8fee-8c209225036c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:52.653664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da5ec9b-1d0e-4d79-8fee-8c209225036c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:52.654019 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da5ec9b-1d0e-4d79-8fee-8c209225036c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:52.654298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da5ec9b-1d0e-4d79-8fee-8c209225036c None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:52.660618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da5ec9b-1d0e-4d79-8fee-8c209225036c None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:52.662272 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da5ec9b-1d0e-4d79-8fee-8c209225036c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:52.664310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 75/156] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:52 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:52.670991 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da5ec9b-1d0e-4d79-8fee-8c209225036c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:52.802828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 82/157] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:52.804515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a4a264-5b4a-41b3-b2e4-4d74d4078376 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:52.805039 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a4a264-5b4a-41b3-b2e4-4d74d4078376 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:52.805674 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a4a264-5b4a-41b3-b2e4-4d74d4078376 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:52.807259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 83/158] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:52.813208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6568a82d-31a7-4712-b52e-66045e46fb01 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:52.813812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6568a82d-31a7-4712-b52e-66045e46fb01 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:52.814290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6568a82d-31a7-4712-b52e-66045e46fb01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:52.827383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da5ec9b-1d0e-4d79-8fee-8c209225036c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:52.851593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6568a82d-31a7-4712-b52e-66045e46fb01 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:52.853137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6568a82d-31a7-4712-b52e-66045e46fb01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:52.869793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:52.884210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:52.897813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6096d2a-8156-4188-8fac-af6ca0cd28ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:52.898685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6096d2a-8156-4188-8fac-af6ca0cd28ba None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:52.899039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6096d2a-8156-4188-8fac-af6ca0cd28ba None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:52.899512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6096d2a-8156-4188-8fac-af6ca0cd28ba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:52.903458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6096d2a-8156-4188-8fac-af6ca0cd28ba None admin] RBAC: Adding query filter params (role.id=ac68e1820c604122981fd5c4e2ebc447, scope.project.id=c0935b3a5a454c70bf75163763e11b34, user.id=a614cb9f929748ecb70ea9ce0aed9916) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:52.903458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6096d2a-8156-4188-8fac-af6ca0cd28ba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:52.903743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6096d2a-8156-4188-8fac-af6ca0cd28ba None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:52.916908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 76/159] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:53:52 2022] GET /identity/v3/role_assignments?user.id=a614cb9f929748ecb70ea9ce0aed9916&scope.project.id=c0935b3a5a454c70bf75163763e11b34&role.id=ac68e1820c604122981fd5c4e2ebc447 => generated 608 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:52.917551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6096d2a-8156-4188-8fac-af6ca0cd28ba None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:53.037264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 84/160] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:53:53.111715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6096d2a-8156-4188-8fac-af6ca0cd28ba None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:53.127629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6568a82d-31a7-4712-b52e-66045e46fb01 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:53.180404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05396f78-b419-47b4-ac67-64b09d7fd8fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:53.180999 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05396f78-b419-47b4-ac67-64b09d7fd8fc None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:53.181278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05396f78-b419-47b4-ac67-64b09d7fd8fc None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:53.181546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05396f78-b419-47b4-ac67-64b09d7fd8fc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:53.201713 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05396f78-b419-47b4-ac67-64b09d7fd8fc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-05396f78-b419-47b4-ac67-64b09d7fd8fc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:53.207563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:53.208481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:53:53.210376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 77/161] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:53:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:53.230356 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05396f78-b419-47b4-ac67-64b09d7fd8fc None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:53.318772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:53.366180 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6568a82d-31a7-4712-b52e-66045e46fb01 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:53.442335 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51247287-0152-4d4d-820d-1556fa409645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:53.443451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51247287-0152-4d4d-820d-1556fa409645 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:53.443877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51247287-0152-4d4d-820d-1556fa409645 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:53.444308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51247287-0152-4d4d-820d-1556fa409645 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:53.451337 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05396f78-b419-47b4-ac67-64b09d7fd8fc None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:53.470341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51247287-0152-4d4d-820d-1556fa409645 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-51247287-0152-4d4d-820d-1556fa409645 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:53.483104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:53.484256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:53.485105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:53.485105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:53.485105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:53.485105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:53.485105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:53.485105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:53.485105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:53.485105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:53.485105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:53.485105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:53.485105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:53:53.485105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:53:53.490316 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 85/162] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 363 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:53.495159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51247287-0152-4d4d-820d-1556fa409645 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:53.500604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:53.543048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cd9c29-83a9-4905-a81a-5cdc55bba071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:53.543983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cd9c29-83a9-4905-a81a-5cdc55bba071 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:53.544420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cd9c29-83a9-4905-a81a-5cdc55bba071 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:53.544854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cd9c29-83a9-4905-a81a-5cdc55bba071 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:53.545767 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cd9c29-83a9-4905-a81a-5cdc55bba071 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:53.546301 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cd9c29-83a9-4905-a81a-5cdc55bba071 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:53.547952 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cd9c29-83a9-4905-a81a-5cdc55bba071 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:53.559456 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 78/163] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:53.567805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cd9c29-83a9-4905-a81a-5cdc55bba071 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:53.733221 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cd9c29-83a9-4905-a81a-5cdc55bba071 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:53.745759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:53.755171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51247287-0152-4d4d-820d-1556fa409645 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:53.782391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2299647a-da9d-48f7-97dc-b2acf3c2defe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:53.783480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2299647a-da9d-48f7-97dc-b2acf3c2defe None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:53.784534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2299647a-da9d-48f7-97dc-b2acf3c2defe None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:53.785698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2299647a-da9d-48f7-97dc-b2acf3c2defe None admin] PATH_INFO: `/v3/users/e711c7fc76434a0b86b92dc8bcf69e87` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:53.787477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2299647a-da9d-48f7-97dc-b2acf3c2defe None admin] RBAC: Authorizing `identity:get_user(user_id=e711c7fc76434a0b86b92dc8bcf69e87)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:53.789311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2299647a-da9d-48f7-97dc-b2acf3c2defe None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:53.791384 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 79/164] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:53 2022] GET /identity/v3/users/e711c7fc76434a0b86b92dc8bcf69e87 => generated 247 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:53.797346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2299647a-da9d-48f7-97dc-b2acf3c2defe None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:53.831466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74416ba3-2b81-4fcb-a72e-c6123681aa58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:53.832606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74416ba3-2b81-4fcb-a72e-c6123681aa58 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:53.833076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74416ba3-2b81-4fcb-a72e-c6123681aa58 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:53.833533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74416ba3-2b81-4fcb-a72e-c6123681aa58 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:53.834428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74416ba3-2b81-4fcb-a72e-c6123681aa58 None admin] RBAC: Adding query filter params (name=member) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:53.834908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74416ba3-2b81-4fcb-a72e-c6123681aa58 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:53.836971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74416ba3-2b81-4fcb-a72e-c6123681aa58 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:53.848086 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 86/165] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:53:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:53.856205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74416ba3-2b81-4fcb-a72e-c6123681aa58 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:53.938130 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:53.995679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74416ba3-2b81-4fcb-a72e-c6123681aa58 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:54.053852 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bfd6a6-a8c1-4405-a448-1abcaefb447c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:54.054962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bfd6a6-a8c1-4405-a448-1abcaefb447c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:54.055539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bfd6a6-a8c1-4405-a448-1abcaefb447c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:54.056010 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bfd6a6-a8c1-4405-a448-1abcaefb447c None admin] PATH_INFO: `/v3/users/a614cb9f929748ecb70ea9ce0aed9916` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:54.060123 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bfd6a6-a8c1-4405-a448-1abcaefb447c None admin] RBAC: Authorizing `identity:get_user(user_id=a614cb9f929748ecb70ea9ce0aed9916)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:54.061983 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bfd6a6-a8c1-4405-a448-1abcaefb447c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:54.063915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 87/166] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:53 2022] GET /identity/v3/users/a614cb9f929748ecb70ea9ce0aed9916 => generated 275 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:54.070044 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bfd6a6-a8c1-4405-a448-1abcaefb447c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:54.104727 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2299647a-da9d-48f7-97dc-b2acf3c2defe None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:54.176225 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764621c1-03af-42c3-b2de-c555a506e432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:54.177276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764621c1-03af-42c3-b2de-c555a506e432 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:54.177551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764621c1-03af-42c3-b2de-c555a506e432 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:54.177958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764621c1-03af-42c3-b2de-c555a506e432 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:54.255020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bfd6a6-a8c1-4405-a448-1abcaefb447c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:54.291787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:54.340477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:54.348881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400fbff1-6a74-498f-9d32-7129ba0dc6bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:54.349557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400fbff1-6a74-498f-9d32-7129ba0dc6bc None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:54.349836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400fbff1-6a74-498f-9d32-7129ba0dc6bc None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:54.350111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400fbff1-6a74-498f-9d32-7129ba0dc6bc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:54.359030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764621c1-03af-42c3-b2de-c555a506e432 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:54.370509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764621c1-03af-42c3-b2de-c555a506e432 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:54.491344 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400fbff1-6a74-498f-9d32-7129ba0dc6bc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:54.498460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400fbff1-6a74-498f-9d32-7129ba0dc6bc None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:54.576114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 80/167] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 779 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:53:54.582392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764621c1-03af-42c3-b2de-c555a506e432 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:54.735340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 88/168] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 665 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:53:54.737804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2217a842-c56c-4720-aad7-1e26ff8fdc6a None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:54.738315 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2217a842-c56c-4720-aad7-1e26ff8fdc6a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:54.738787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2217a842-c56c-4720-aad7-1e26ff8fdc6a None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:54.740422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:54.740643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 89/169] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:54.741822 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2217a842-c56c-4720-aad7-1e26ff8fdc6a None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:54.776473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:54.807973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764621c1-03af-42c3-b2de-c555a506e432 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:54.864528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2217a842-c56c-4720-aad7-1e26ff8fdc6a None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:54.891488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c75337-e951-4234-af48-78c6ee0ed224 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:54.892157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c75337-e951-4234-af48-78c6ee0ed224 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:54.892437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c75337-e951-4234-af48-78c6ee0ed224 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:54.892713 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c75337-e951-4234-af48-78c6ee0ed224 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:54.894083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c75337-e951-4234-af48-78c6ee0ed224 None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:54.895748 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c75337-e951-4234-af48-78c6ee0ed224 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:54.907381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 81/170] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:54 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:53:54.907381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11262b96-3be3-499a-bab9-de97d7c0c4b8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:54.907381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11262b96-3be3-499a-bab9-de97d7c0c4b8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:54.907381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11262b96-3be3-499a-bab9-de97d7c0c4b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:54.935055 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8f6ea7-c6a2-47ff-b436-6b1e46e7d165 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:54.936235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8f6ea7-c6a2-47ff-b436-6b1e46e7d165 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:54.936741 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8f6ea7-c6a2-47ff-b436-6b1e46e7d165 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:54.937037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-11262b96-3be3-499a-bab9-de97d7c0c4b8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:54.937462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8f6ea7-c6a2-47ff-b436-6b1e46e7d165 None admin] PATH_INFO: `/v3/projects/9702636a96e94d8c8dbfe37dd9170280` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:54.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-11262b96-3be3-499a-bab9-de97d7c0c4b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:54.939136 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8f6ea7-c6a2-47ff-b436-6b1e46e7d165 None admin] RBAC: Authorizing `identity:get_project(project_id=9702636a96e94d8c8dbfe37dd9170280)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:54.940991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8f6ea7-c6a2-47ff-b436-6b1e46e7d165 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:54.943327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 90/171] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:54 2022] GET /identity/v3/projects/9702636a96e94d8c8dbfe37dd9170280 => generated 312 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:54.948776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8f6ea7-c6a2-47ff-b436-6b1e46e7d165 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:55.078765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:55.084448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8f6ea7-c6a2-47ff-b436-6b1e46e7d165 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:55.104358 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 82/172] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:55.112699 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11262b96-3be3-499a-bab9-de97d7c0c4b8 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:55.131390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c93942-05a7-49b7-a6da-0c4b690dcdbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:55.134485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c93942-05a7-49b7-a6da-0c4b690dcdbf None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:55.135187 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c93942-05a7-49b7-a6da-0c4b690dcdbf None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:55.135926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c93942-05a7-49b7-a6da-0c4b690dcdbf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:55.139546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c93942-05a7-49b7-a6da-0c4b690dcdbf None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=c0935b3a5a454c70bf75163763e11b34, user.id=e711c7fc76434a0b86b92dc8bcf69e87) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:55.140451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c93942-05a7-49b7-a6da-0c4b690dcdbf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:55.144040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c93942-05a7-49b7-a6da-0c4b690dcdbf None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:55.158081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 91/173] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:53:54 2022] GET /identity/v3/role_assignments?user.id=e711c7fc76434a0b86b92dc8bcf69e87&scope.project.id=c0935b3a5a454c70bf75163763e11b34&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 608 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:55.159961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c93942-05a7-49b7-a6da-0c4b690dcdbf None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:55.295613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:55.351457 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11262b96-3be3-499a-bab9-de97d7c0c4b8 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:55.371421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c93942-05a7-49b7-a6da-0c4b690dcdbf None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:55.413520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcd8bac-d6a6-46fb-bc84-ca3ba1d49801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:55.414537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcd8bac-d6a6-46fb-bc84-ca3ba1d49801 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:55.415008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcd8bac-d6a6-46fb-bc84-ca3ba1d49801 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:55.415524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcd8bac-d6a6-46fb-bc84-ca3ba1d49801 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:55.416666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcd8bac-d6a6-46fb-bc84-ca3ba1d49801 None admin] RBAC: Adding query filter params (role.id=ac68e1820c604122981fd5c4e2ebc447, scope.project.id=9702636a96e94d8c8dbfe37dd9170280, user.id=a614cb9f929748ecb70ea9ce0aed9916) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:55.417077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcd8bac-d6a6-46fb-bc84-ca3ba1d49801 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:55.419829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcd8bac-d6a6-46fb-bc84-ca3ba1d49801 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:55.430217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 83/174] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:53:55 2022] GET /identity/v3/role_assignments?user.id=a614cb9f929748ecb70ea9ce0aed9916&scope.project.id=9702636a96e94d8c8dbfe37dd9170280&role.id=ac68e1820c604122981fd5c4e2ebc447 => generated 608 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:55.453111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf2a05b-a110-4439-8ed0-177ace80399d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:55.453919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf2a05b-a110-4439-8ed0-177ace80399d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:55.454210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf2a05b-a110-4439-8ed0-177ace80399d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:55.454491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf2a05b-a110-4439-8ed0-177ace80399d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:55.474854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf2a05b-a110-4439-8ed0-177ace80399d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-eaf2a05b-a110-4439-8ed0-177ace80399d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:55.480756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:55.481706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:53:55.482480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 92/175] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:53:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:55.488873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcd8bac-d6a6-46fb-bc84-ca3ba1d49801 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:55.719322 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcd8bac-d6a6-46fb-bc84-ca3ba1d49801 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:55.720806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1734b35-d0c5-4b85-8107-6d4157c45183 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:55.721083 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1734b35-d0c5-4b85-8107-6d4157c45183 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:55.721932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1734b35-d0c5-4b85-8107-6d4157c45183 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:55.723339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 93/176] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:55.730144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f2b429-20af-4fd0-9a45-8f0b5c4e7b09 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:55.730449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f2b429-20af-4fd0-9a45-8f0b5c4e7b09 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:55.730709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f2b429-20af-4fd0-9a45-8f0b5c4e7b09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:55.785735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-16f2b429-20af-4fd0-9a45-8f0b5c4e7b09 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:55.787817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-16f2b429-20af-4fd0-9a45-8f0b5c4e7b09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:55.793811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78700a04-cf9d-4b41-8fb0-a3085194a802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:55.794451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78700a04-cf9d-4b41-8fb0-a3085194a802 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:55.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78700a04-cf9d-4b41-8fb0-a3085194a802 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:55.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78700a04-cf9d-4b41-8fb0-a3085194a802 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:55.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78700a04-cf9d-4b41-8fb0-a3085194a802 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:55.796211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78700a04-cf9d-4b41-8fb0-a3085194a802 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:55.797498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78700a04-cf9d-4b41-8fb0-a3085194a802 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:55.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:55.815553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 84/177] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:53:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:55.817639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b18da66-8713-49a8-8e61-4713a40d319b None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:55.818203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b18da66-8713-49a8-8e61-4713a40d319b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:55.818675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b18da66-8713-49a8-8e61-4713a40d319b None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:55.821237 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 85/178] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:55 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:55.824911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b18da66-8713-49a8-8e61-4713a40d319b None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:55.836878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:55.955379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 94/179] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:53:55.955379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11045036-abff-4670-a0aa-c18413287700 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:55.955379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11045036-abff-4670-a0aa-c18413287700 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:55.955379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11045036-abff-4670-a0aa-c18413287700 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:55.981474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-11045036-abff-4670-a0aa-c18413287700 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:55.983117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-11045036-abff-4670-a0aa-c18413287700 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:56.013658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b18da66-8713-49a8-8e61-4713a40d319b None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:56.073220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e863f7-fe0e-474f-a904-50fec18641a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:56.073953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e863f7-fe0e-474f-a904-50fec18641a9 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:56.074378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e863f7-fe0e-474f-a904-50fec18641a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:56.074721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e863f7-fe0e-474f-a904-50fec18641a9 None admin] PATH_INFO: `/v3/users/a614cb9f929748ecb70ea9ce0aed9916` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:56.076126 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e863f7-fe0e-474f-a904-50fec18641a9 None admin] RBAC: Authorizing `identity:get_user(user_id=a614cb9f929748ecb70ea9ce0aed9916)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:56.077827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e863f7-fe0e-474f-a904-50fec18641a9 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:56.079757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 86/180] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:55 2022] GET /identity/v3/users/a614cb9f929748ecb70ea9ce0aed9916 => generated 275 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:56.081736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67098523-1f67-4dbd-ac1e-23a2ea37e2ef None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:56.082141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67098523-1f67-4dbd-ac1e-23a2ea37e2ef None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:56.082544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67098523-1f67-4dbd-ac1e-23a2ea37e2ef None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:56.084286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 87/181] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:56.086251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5f6e3c-2bfe-4f5f-b19d-2e7f751f99e6 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:56.086668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5f6e3c-2bfe-4f5f-b19d-2e7f751f99e6 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:56.087006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5f6e3c-2bfe-4f5f-b19d-2e7f751f99e6 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:56.089162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 88/182] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:53:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:53:56.090446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5f6e3c-2bfe-4f5f-b19d-2e7f751f99e6 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:56.101343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 95/183] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:56.102720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11045036-abff-4670-a0aa-c18413287700 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:56.211293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:56.215808 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:56.236555 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11045036-abff-4670-a0aa-c18413287700 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:56.264347 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5f6e3c-2bfe-4f5f-b19d-2e7f751f99e6 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:56.282645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fceba8c5-145b-46b3-bd5e-6b81b367440c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:56.283277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fceba8c5-145b-46b3-bd5e-6b81b367440c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:56.283582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fceba8c5-145b-46b3-bd5e-6b81b367440c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:56.283886 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fceba8c5-145b-46b3-bd5e-6b81b367440c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:56.303988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a7fe5c-8df7-4ec7-9523-95d46cfa3085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:56.304778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a7fe5c-8df7-4ec7-9523-95d46cfa3085 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:56.305262 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a7fe5c-8df7-4ec7-9523-95d46cfa3085 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:56.305571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a7fe5c-8df7-4ec7-9523-95d46cfa3085 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:56.335070 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a7fe5c-8df7-4ec7-9523-95d46cfa3085 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-d1a7fe5c-8df7-4ec7-9523-95d46cfa3085 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:56.341149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:56.342233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:53:56.343038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 89/184] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:53:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:56.344167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8546972e-a25a-4416-8696-5771c2a7e74c None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:56.344167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8546972e-a25a-4416-8696-5771c2a7e74c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:56.344167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8546972e-a25a-4416-8696-5771c2a7e74c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:56.367613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-8546972e-a25a-4416-8696-5771c2a7e74c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:56.368867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8546972e-a25a-4416-8696-5771c2a7e74c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:56.389882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fceba8c5-145b-46b3-bd5e-6b81b367440c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:56.392120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fceba8c5-145b-46b3-bd5e-6b81b367440c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:56.447678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 90/185] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:56.449290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f322bb24-2eac-4ee0-9318-64684ec3a045 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:56.449629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f322bb24-2eac-4ee0-9318-64684ec3a045 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:56.449984 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f322bb24-2eac-4ee0-9318-64684ec3a045 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:56.472541 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f322bb24-2eac-4ee0-9318-64684ec3a045 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:53:56.473746 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f322bb24-2eac-4ee0-9318-64684ec3a045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:53:56.510476 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:56.544883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 96/186] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:53:56.545832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fceba8c5-145b-46b3-bd5e-6b81b367440c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:56.560661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:56.578637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 91/187] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:53:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:53:56.579526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f322bb24-2eac-4ee0-9318-64684ec3a045 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:56.661324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fceba8c5-145b-46b3-bd5e-6b81b367440c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:56.698581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e056092-759b-4363-9f2d-05dfa90da246 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:56.699231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e056092-759b-4363-9f2d-05dfa90da246 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:56.699542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e056092-759b-4363-9f2d-05dfa90da246 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:56.699820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e056092-759b-4363-9f2d-05dfa90da246 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:56.711242 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e056092-759b-4363-9f2d-05dfa90da246 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:56.712314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f322bb24-2eac-4ee0-9318-64684ec3a045 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-2e056092-759b-4363-9f2d-05dfa90da246 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:56.716264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:56.719901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 97/188] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:53:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:56.721069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e056092-759b-4363-9f2d-05dfa90da246 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:56.751951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e3a1c3-2d2d-4b99-97ab-e774771fa294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:56.753089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e3a1c3-2d2d-4b99-97ab-e774771fa294 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:56.753582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e3a1c3-2d2d-4b99-97ab-e774771fa294 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:56.754081 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e3a1c3-2d2d-4b99-97ab-e774771fa294 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:56.761534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e3a1c3-2d2d-4b99-97ab-e774771fa294 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:56.761534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e3a1c3-2d2d-4b99-97ab-e774771fa294 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:56.761534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e3a1c3-2d2d-4b99-97ab-e774771fa294 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:56.770365 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 92/189] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:56.770365 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e3a1c3-2d2d-4b99-97ab-e774771fa294 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:56.787256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:56.836519 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:56.858297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e056092-759b-4363-9f2d-05dfa90da246 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:56.896982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e3a1c3-2d2d-4b99-97ab-e774771fa294 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:56.915074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00c5252-7b96-4868-ae91-3dfe9498babf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:56.916199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00c5252-7b96-4868-ae91-3dfe9498babf None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:56.916687 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00c5252-7b96-4868-ae91-3dfe9498babf None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:56.917154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00c5252-7b96-4868-ae91-3dfe9498babf None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:56.931985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe06a4d3-83d3-46b8-bae7-da1a4ca666ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:56.932721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe06a4d3-83d3-46b8-bae7-da1a4ca666ea None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:56.933100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe06a4d3-83d3-46b8-bae7-da1a4ca666ea None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:56.933448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe06a4d3-83d3-46b8-bae7-da1a4ca666ea None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:56.933673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00c5252-7b96-4868-ae91-3dfe9498babf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:53:56.934765 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe06a4d3-83d3-46b8-bae7-da1a4ca666ea None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:56.936330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe06a4d3-83d3-46b8-bae7-da1a4ca666ea None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:56.938004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 93/190] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:56 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:56.940040 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe06a4d3-83d3-46b8-bae7-da1a4ca666ea None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-f00c5252-7b96-4868-ae91-3dfe9498babf None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:53:56.941262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:53:56.942117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:53:56.942921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:53:56.942921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:53:56.942921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:53:56.942921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:53:56.942921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:53:56.942921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:53:56.942921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:53:56.942921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:53:56.942921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:53:56.942921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:53:56.942921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:53:56.942921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:53:56.943604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 98/191] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:53:56 2022] GET /identity/v3/roles/reader => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:53:56.944181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00c5252-7b96-4868-ae91-3dfe9498babf None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:57.063957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe06a4d3-83d3-46b8-bae7-da1a4ca666ea None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:57.071464 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00c5252-7b96-4868-ae91-3dfe9498babf None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:57.074301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:57.085921 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:57.111583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2019dbae-27ae-45ab-8582-06bb3f435443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:57.113312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2019dbae-27ae-45ab-8582-06bb3f435443 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:57.113794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2019dbae-27ae-45ab-8582-06bb3f435443 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:57.114233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2019dbae-27ae-45ab-8582-06bb3f435443 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:57.116326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2019dbae-27ae-45ab-8582-06bb3f435443 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:57.119084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2019dbae-27ae-45ab-8582-06bb3f435443 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:57.122069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 94/192] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:53:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:57.123533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2019dbae-27ae-45ab-8582-06bb3f435443 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:57.137225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44c7996-c8d6-4d62-ae9d-473c1ad67ae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:57.138174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44c7996-c8d6-4d62-ae9d-473c1ad67ae1 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:57.138631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44c7996-c8d6-4d62-ae9d-473c1ad67ae1 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:57.139182 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44c7996-c8d6-4d62-ae9d-473c1ad67ae1 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:57.140593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44c7996-c8d6-4d62-ae9d-473c1ad67ae1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:57.141091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44c7996-c8d6-4d62-ae9d-473c1ad67ae1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:57.143204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44c7996-c8d6-4d62-ae9d-473c1ad67ae1 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:57.154944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 99/193] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:53:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:57.156026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44c7996-c8d6-4d62-ae9d-473c1ad67ae1 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:57.255823 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2019dbae-27ae-45ab-8582-06bb3f435443 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:57.288526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44c7996-c8d6-4d62-ae9d-473c1ad67ae1 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:57.300053 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44ce02a-7dd4-48cc-95b2-3a80a7b558a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:57.300777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44ce02a-7dd4-48cc-95b2-3a80a7b558a2 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:57.301065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44ce02a-7dd4-48cc-95b2-3a80a7b558a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:57.301334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44ce02a-7dd4-48cc-95b2-3a80a7b558a2 None admin] PATH_INFO: `/v3/users/a1412db69e884863830ad4c27e6e9970` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:57.316841 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44ce02a-7dd4-48cc-95b2-3a80a7b558a2 None admin] RBAC: Authorizing `identity:get_user(user_id=a1412db69e884863830ad4c27e6e9970)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:57.318508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44ce02a-7dd4-48cc-95b2-3a80a7b558a2 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:57.320353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 95/194] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:57 2022] GET /identity/v3/users/a1412db69e884863830ad4c27e6e9970 => generated 283 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:57.321192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44ce02a-7dd4-48cc-95b2-3a80a7b558a2 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:57.341094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded85d3f-156e-4763-a9f7-cdd61e019e0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:57.341790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded85d3f-156e-4763-a9f7-cdd61e019e0a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:57.342554 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded85d3f-156e-4763-a9f7-cdd61e019e0a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:57.342554 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded85d3f-156e-4763-a9f7-cdd61e019e0a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:57.343288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded85d3f-156e-4763-a9f7-cdd61e019e0a None admin] RBAC: Adding query filter params (role.id=0a2f5ca94533412088499eaccd7b2e21, scope.project.id=c0935b3a5a454c70bf75163763e11b34, user.id=a614cb9f929748ecb70ea9ce0aed9916) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:57.343623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded85d3f-156e-4763-a9f7-cdd61e019e0a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:57.345950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded85d3f-156e-4763-a9f7-cdd61e019e0a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:57.355457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 100/195] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:53:57 2022] GET /identity/v3/role_assignments?user.id=a614cb9f929748ecb70ea9ce0aed9916&scope.project.id=c0935b3a5a454c70bf75163763e11b34&role.id=0a2f5ca94533412088499eaccd7b2e21 => generated 608 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:57.357743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded85d3f-156e-4763-a9f7-cdd61e019e0a None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:57.391995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:57.397460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:57.494687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44ce02a-7dd4-48cc-95b2-3a80a7b558a2 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:57.529497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded85d3f-156e-4763-a9f7-cdd61e019e0a None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:57.559687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addc851e-732a-498e-83f2-f25e93549311 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:57.560454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addc851e-732a-498e-83f2-f25e93549311 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:57.560868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addc851e-732a-498e-83f2-f25e93549311 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:57.561217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addc851e-732a-498e-83f2-f25e93549311 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:57.562275 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addc851e-732a-498e-83f2-f25e93549311 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:57.562668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addc851e-732a-498e-83f2-f25e93549311 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:57.564340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addc851e-732a-498e-83f2-f25e93549311 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:57.580574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 96/196] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:53:57 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:57.580574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addc851e-732a-498e-83f2-f25e93549311 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:57.591463 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c75b0a-c705-4c21-aa01-14bb66be36f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:57.591839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c75b0a-c705-4c21-aa01-14bb66be36f2 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:57.592251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c75b0a-c705-4c21-aa01-14bb66be36f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:57.592704 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c75b0a-c705-4c21-aa01-14bb66be36f2 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:57.593606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c75b0a-c705-4c21-aa01-14bb66be36f2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:57.595808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c75b0a-c705-4c21-aa01-14bb66be36f2 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:57.629663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 101/197] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:53:57 2022] POST /identity/v3/users => generated 297 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:53:57.630877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c75b0a-c705-4c21-aa01-14bb66be36f2 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:57.746888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addc851e-732a-498e-83f2-f25e93549311 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:57.761075 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:57.767974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:57.796805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c75b0a-c705-4c21-aa01-14bb66be36f2 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:57.811651 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b891a87-64c3-425f-bb54-5e6c43bd0893 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:57.813578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b891a87-64c3-425f-bb54-5e6c43bd0893 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:57.813578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b891a87-64c3-425f-bb54-5e6c43bd0893 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:57.813578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b891a87-64c3-425f-bb54-5e6c43bd0893 None admin] PATH_INFO: `/v3/users/a1412db69e884863830ad4c27e6e9970` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:57.814932 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b891a87-64c3-425f-bb54-5e6c43bd0893 None admin] RBAC: Authorizing `identity:get_user(user_id=a1412db69e884863830ad4c27e6e9970)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:57.816515 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b891a87-64c3-425f-bb54-5e6c43bd0893 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:57.818237 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 97/198] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:57 2022] GET /identity/v3/users/a1412db69e884863830ad4c27e6e9970 => generated 283 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:57.819020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b891a87-64c3-425f-bb54-5e6c43bd0893 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:57.843451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd96f1df-47a4-4025-98f3-0c7bf2219c58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:57.843981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd96f1df-47a4-4025-98f3-0c7bf2219c58 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:57.844256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd96f1df-47a4-4025-98f3-0c7bf2219c58 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:57.844528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd96f1df-47a4-4025-98f3-0c7bf2219c58 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:57.952408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd96f1df-47a4-4025-98f3-0c7bf2219c58 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:57.954541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd96f1df-47a4-4025-98f3-0c7bf2219c58 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:57.973395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b891a87-64c3-425f-bb54-5e6c43bd0893 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:58.022395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a77756-1a91-4566-be90-c79ea6be2cd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:58.023058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a77756-1a91-4566-be90-c79ea6be2cd5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:58.023342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a77756-1a91-4566-be90-c79ea6be2cd5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:58.023652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a77756-1a91-4566-be90-c79ea6be2cd5 None admin] PATH_INFO: `/v3/users/acdd051bae9b454c99a875a522cdc828` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:58.024984 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a77756-1a91-4566-be90-c79ea6be2cd5 None admin] RBAC: Authorizing `identity:get_user(user_id=acdd051bae9b454c99a875a522cdc828)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:58.026899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a77756-1a91-4566-be90-c79ea6be2cd5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:58.029358 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 98/199] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:53:57 2022] GET /identity/v3/users/acdd051bae9b454c99a875a522cdc828 => generated 289 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:58.030271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a77756-1a91-4566-be90-c79ea6be2cd5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:58.102104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:58.125616 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 102/200] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:53:58.126774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd96f1df-47a4-4025-98f3-0c7bf2219c58 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:58.147637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:58.220851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a77756-1a91-4566-be90-c79ea6be2cd5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:58.271942 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce6bb35-e8bf-4231-9b2d-7531616c111f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:58.272566 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce6bb35-e8bf-4231-9b2d-7531616c111f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:58.272853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce6bb35-e8bf-4231-9b2d-7531616c111f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:58.273145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce6bb35-e8bf-4231-9b2d-7531616c111f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:58.302442 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd96f1df-47a4-4025-98f3-0c7bf2219c58 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:58.367858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b412446-01aa-4989-8dd5-4834b90b4c19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:58.369568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b412446-01aa-4989-8dd5-4834b90b4c19 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:58.370260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b412446-01aa-4989-8dd5-4834b90b4c19 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:58.370743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b412446-01aa-4989-8dd5-4834b90b4c19 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:58.386876 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce6bb35-e8bf-4231-9b2d-7531616c111f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:58.390205 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce6bb35-e8bf-4231-9b2d-7531616c111f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:58.462619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:58.471244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:58.485951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b412446-01aa-4989-8dd5-4834b90b4c19 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:58.488208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b412446-01aa-4989-8dd5-4834b90b4c19 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:58.581343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 99/201] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 551 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:53:58.583020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce6bb35-e8bf-4231-9b2d-7531616c111f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:58.649079 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 103/202] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:53:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:53:58.654574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b412446-01aa-4989-8dd5-4834b90b4c19 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:58.741396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce6bb35-e8bf-4231-9b2d-7531616c111f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:58.770295 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:58.792257 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b412446-01aa-4989-8dd5-4834b90b4c19 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:58.794683 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ca696b-f621-4071-b244-50705c005518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:58.795970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ca696b-f621-4071-b244-50705c005518 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:58.796443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ca696b-f621-4071-b244-50705c005518 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:58.796980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ca696b-f621-4071-b244-50705c005518 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:58.798332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ca696b-f621-4071-b244-50705c005518 None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:58.800129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ca696b-f621-4071-b244-50705c005518 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:58.802674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 100/203] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:58 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:58.804074 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ca696b-f621-4071-b244-50705c005518 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:58.818748 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:58.837934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d9ba34-a902-4e07-af4b-0561d87676ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:58.839068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d9ba34-a902-4e07-af4b-0561d87676ff None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:58.839680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d9ba34-a902-4e07-af4b-0561d87676ff None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:58.840176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d9ba34-a902-4e07-af4b-0561d87676ff None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:58.841699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d9ba34-a902-4e07-af4b-0561d87676ff None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:58.843828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d9ba34-a902-4e07-af4b-0561d87676ff None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:58.846172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 104/204] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:58 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:58.848006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d9ba34-a902-4e07-af4b-0561d87676ff None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:58.966322 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ca696b-f621-4071-b244-50705c005518 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:59.006349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76dfc25-08c2-401a-b502-a2d265700436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:59.007012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76dfc25-08c2-401a-b502-a2d265700436 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:59.007295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76dfc25-08c2-401a-b502-a2d265700436 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:59.007637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76dfc25-08c2-401a-b502-a2d265700436 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:59.009089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76dfc25-08c2-401a-b502-a2d265700436 None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:59.010856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76dfc25-08c2-401a-b502-a2d265700436 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:59.013009 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 101/205] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:53:58 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:59.013708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76dfc25-08c2-401a-b502-a2d265700436 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:59.019677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d9ba34-a902-4e07-af4b-0561d87676ff None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:59.090525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8539e18a-423b-44ae-9b8d-c6a7f12b48c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:59.091379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539e18a-423b-44ae-9b8d-c6a7f12b48c3 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:59.092039 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539e18a-423b-44ae-9b8d-c6a7f12b48c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:59.092786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539e18a-423b-44ae-9b8d-c6a7f12b48c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:59.093842 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8539e18a-423b-44ae-9b8d-c6a7f12b48c3 None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=2e832f037cf845379232686bfc78f98d, user.id=a1412db69e884863830ad4c27e6e9970) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:59.094787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8539e18a-423b-44ae-9b8d-c6a7f12b48c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:59.097273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8539e18a-423b-44ae-9b8d-c6a7f12b48c3 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:59.105983 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 105/206] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:53:58 2022] GET /identity/v3/role_assignments?user.id=a1412db69e884863830ad4c27e6e9970&scope.project.id=2e832f037cf845379232686bfc78f98d&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 259 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:59.107563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8539e18a-423b-44ae-9b8d-c6a7f12b48c3 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:53:59.151506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:59.153283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:53:59.202100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76dfc25-08c2-401a-b502-a2d265700436 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:59.249856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0be2ecd-fa34-44ec-8398-4a5127da7290 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:59.250409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0be2ecd-fa34-44ec-8398-4a5127da7290 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:59.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0be2ecd-fa34-44ec-8398-4a5127da7290 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:59.251079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0be2ecd-fa34-44ec-8398-4a5127da7290 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:59.252066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0be2ecd-fa34-44ec-8398-4a5127da7290 None admin] RBAC: Adding query filter params (role.id=0a2f5ca94533412088499eaccd7b2e21, scope.project.id=2e832f037cf845379232686bfc78f98d, user.id=a1412db69e884863830ad4c27e6e9970) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:59.252910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0be2ecd-fa34-44ec-8398-4a5127da7290 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:59.254948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0be2ecd-fa34-44ec-8398-4a5127da7290 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:59.263159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 102/207] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:53:59 2022] GET /identity/v3/role_assignments?user.id=a1412db69e884863830ad4c27e6e9970&scope.project.id=2e832f037cf845379232686bfc78f98d&role.id=0a2f5ca94533412088499eaccd7b2e21 => generated 259 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:53:59.363683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8539e18a-423b-44ae-9b8d-c6a7f12b48c3 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:53:59.415183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e304afd7-acea-414a-a878-06b2cae9efef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:53:59.415878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e304afd7-acea-414a-a878-06b2cae9efef None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:53:59.416163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e304afd7-acea-414a-a878-06b2cae9efef None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:53:59.416441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e304afd7-acea-414a-a878-06b2cae9efef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:53:59.417254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e304afd7-acea-414a-a878-06b2cae9efef None admin] RBAC: Adding query filter params (role.id=de6529a9de0b48b2910599a3a56124f3, scope.project.id=c0935b3a5a454c70bf75163763e11b34, user.id=acdd051bae9b454c99a875a522cdc828) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:53:59.417570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e304afd7-acea-414a-a878-06b2cae9efef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:53:59.419156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e304afd7-acea-414a-a878-06b2cae9efef None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:53:59.426628 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 106/208] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:53:59 2022] GET /identity/v3/role_assignments?user.id=acdd051bae9b454c99a875a522cdc828&scope.project.id=c0935b3a5a454c70bf75163763e11b34&role.id=de6529a9de0b48b2910599a3a56124f3 => generated 608 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:00.309799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408288b4-9939-46de-a438-a25efe7a3929 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:00.310329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408288b4-9939-46de-a438-a25efe7a3929 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:00.310329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408288b4-9939-46de-a438-a25efe7a3929 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:00.311746 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 103/209] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:00.319904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924cf30b-883f-410e-aeda-aeb6e281a1a4 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:00.320897 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924cf30b-883f-410e-aeda-aeb6e281a1a4 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:00.321370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924cf30b-883f-410e-aeda-aeb6e281a1a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:00.348288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-924cf30b-883f-410e-aeda-aeb6e281a1a4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:00.350560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-924cf30b-883f-410e-aeda-aeb6e281a1a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:00.370778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8c2f2f-d297-4e3d-9ec2-6988548db5e2 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:00.371167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8c2f2f-d297-4e3d-9ec2-6988548db5e2 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:00.371642 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8c2f2f-d297-4e3d-9ec2-6988548db5e2 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:00.374018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 104/210] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:00 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:54:00.381882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff0e5ca-4759-451d-b723-6b8116a0f228 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:00.382349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff0e5ca-4759-451d-b723-6b8116a0f228 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:00.382840 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff0e5ca-4759-451d-b723-6b8116a0f228 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:00.411710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-8ff0e5ca-4759-451d-b723-6b8116a0f228 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:00.413667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8ff0e5ca-4759-451d-b723-6b8116a0f228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:00.447644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 107/211] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:00.476081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924cf30b-883f-410e-aeda-aeb6e281a1a4 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:00.495219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:00.542447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 105/212] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:54:00.576770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff0e5ca-4759-451d-b723-6b8116a0f228 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:00.606548 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924cf30b-883f-410e-aeda-aeb6e281a1a4 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:00.651563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de4a3f5-dee6-4b30-8e70-c6aad09ac603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:00.652425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de4a3f5-dee6-4b30-8e70-c6aad09ac603 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:00.652803 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de4a3f5-dee6-4b30-8e70-c6aad09ac603 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:00.653926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de4a3f5-dee6-4b30-8e70-c6aad09ac603 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:00.657545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de4a3f5-dee6-4b30-8e70-c6aad09ac603 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:00.661810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de4a3f5-dee6-4b30-8e70-c6aad09ac603 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:00.664737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 108/213] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:00.675165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de4a3f5-dee6-4b30-8e70-c6aad09ac603 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:00.680091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:00.724816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff0e5ca-4759-451d-b723-6b8116a0f228 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:00.776775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:00.786277 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7630386e-dd9b-4ceb-bf6f-540a18a9755b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:00.788044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630386e-dd9b-4ceb-bf6f-540a18a9755b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:00.788511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630386e-dd9b-4ceb-bf6f-540a18a9755b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:00.788947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630386e-dd9b-4ceb-bf6f-540a18a9755b None admin] PATH_INFO: `/v3/roles/member` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:00.804931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de4a3f5-dee6-4b30-8e70-c6aad09ac603 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:00.812908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7630386e-dd9b-4ceb-bf6f-540a18a9755b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-7630386e-dd9b-4ceb-bf6f-540a18a9755b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:00.825229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:00.827033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:00.828762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:00.828762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:00.828762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:00.828762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:00.828762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:00.828762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:00.828762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:00.828762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:00.828762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:00.828762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:00.828762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:00.828762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:00.839859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 106/214] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:00.840252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7630386e-dd9b-4ceb-bf6f-540a18a9755b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:00.857256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c06f598-134d-4ab5-9468-ab625a9d661b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:00.857939 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c06f598-134d-4ab5-9468-ab625a9d661b None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:00.858219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c06f598-134d-4ab5-9468-ab625a9d661b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:00.858494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c06f598-134d-4ab5-9468-ab625a9d661b None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:00.859235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c06f598-134d-4ab5-9468-ab625a9d661b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:00.860908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c06f598-134d-4ab5-9468-ab625a9d661b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:00.887049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 109/215] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:54:00 2022] POST /identity/v3/users => generated 297 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 00:54:01.010438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7630386e-dd9b-4ceb-bf6f-540a18a9755b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:01.050724 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:01.058678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21df362b-a5bf-44d7-89c5-55cbcee56105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:01.059751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21df362b-a5bf-44d7-89c5-55cbcee56105 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:01.060231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21df362b-a5bf-44d7-89c5-55cbcee56105 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:01.060700 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21df362b-a5bf-44d7-89c5-55cbcee56105 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:01.061657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21df362b-a5bf-44d7-89c5-55cbcee56105 None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:01.062176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21df362b-a5bf-44d7-89c5-55cbcee56105 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:01.063899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21df362b-a5bf-44d7-89c5-55cbcee56105 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:01.076060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 107/216] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:01.083168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c06f598-134d-4ab5-9468-ab625a9d661b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:01.233002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c06f598-134d-4ab5-9468-ab625a9d661b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:01.287341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b721d428-2fe9-401f-a201-b2ca35b90cd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:01.288580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b721d428-2fe9-401f-a201-b2ca35b90cd0 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:01.289139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b721d428-2fe9-401f-a201-b2ca35b90cd0 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:01.289619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b721d428-2fe9-401f-a201-b2ca35b90cd0 None admin] PATH_INFO: `/v3/users/d2b15600b8934082a524b413e42230b8` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:01.294099 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:01.311807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b721d428-2fe9-401f-a201-b2ca35b90cd0 None admin] RBAC: Authorizing `identity:get_user(user_id=d2b15600b8934082a524b413e42230b8)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:01.313301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b721d428-2fe9-401f-a201-b2ca35b90cd0 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:01.316054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 110/217] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:01 2022] GET /identity/v3/users/d2b15600b8934082a524b413e42230b8 => generated 297 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:01.322160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21df362b-a5bf-44d7-89c5-55cbcee56105 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:01.472143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21df362b-a5bf-44d7-89c5-55cbcee56105 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:01.530625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb98d26-a648-44ba-a129-70f5ee221e09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:01.531472 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb98d26-a648-44ba-a129-70f5ee221e09 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:01.531757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb98d26-a648-44ba-a129-70f5ee221e09 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:01.532085 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb98d26-a648-44ba-a129-70f5ee221e09 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:01.624541 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:01.635554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb98d26-a648-44ba-a129-70f5ee221e09 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:01.637460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb98d26-a648-44ba-a129-70f5ee221e09 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:01.770117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 108/218] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:01.776568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b721d428-2fe9-401f-a201-b2ca35b90cd0 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:01.941702 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebb06fc-0413-4d69-92ef-24133bffdcda None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:01.942107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebb06fc-0413-4d69-92ef-24133bffdcda None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:01.942328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebb06fc-0413-4d69-92ef-24133bffdcda None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:01.943984 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 109/219] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:54:01.949970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e35f9ba-af0f-4b42-9e93-da49a80af1ed None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:01.950360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e35f9ba-af0f-4b42-9e93-da49a80af1ed None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:01.950601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e35f9ba-af0f-4b42-9e93-da49a80af1ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:01.969744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b721d428-2fe9-401f-a201-b2ca35b90cd0 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:01.990848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-3e35f9ba-af0f-4b42-9e93-da49a80af1ed None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:01.993466 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3e35f9ba-af0f-4b42-9e93-da49a80af1ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:02.068078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db71b7ed-e818-4e39-937d-4e1d21900910 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:02.068784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db71b7ed-e818-4e39-937d-4e1d21900910 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:02.069079 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db71b7ed-e818-4e39-937d-4e1d21900910 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:02.069401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db71b7ed-e818-4e39-937d-4e1d21900910 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:02.071253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db71b7ed-e818-4e39-937d-4e1d21900910 None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:02.073149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db71b7ed-e818-4e39-937d-4e1d21900910 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:02.076328 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 111/220] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:01 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:02.077721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e1b603-44e8-4e7e-be0c-9e9e0085f04f None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:02.078003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e1b603-44e8-4e7e-be0c-9e9e0085f04f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:02.078289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e1b603-44e8-4e7e-be0c-9e9e0085f04f None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:02.084960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 112/221] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:02.086096 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e1b603-44e8-4e7e-be0c-9e9e0085f04f None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:02.089915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 110/222] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:54:02.092834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8628c6c-5abb-4c59-8c27-7293196ec4f8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:02.093344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8628c6c-5abb-4c59-8c27-7293196ec4f8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:02.093843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8628c6c-5abb-4c59-8c27-7293196ec4f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:02.128527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-e8628c6c-5abb-4c59-8c27-7293196ec4f8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:02.129891 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e8628c6c-5abb-4c59-8c27-7293196ec4f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:02.152402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:02.202270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:02.217930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 111/223] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:02.218954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8628c6c-5abb-4c59-8c27-7293196ec4f8 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:02.262841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e1b603-44e8-4e7e-be0c-9e9e0085f04f None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:02.314466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d3106d-cb6d-44e9-bef7-b7720fa3a156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:02.316319 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d3106d-cb6d-44e9-bef7-b7720fa3a156 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:02.317112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d3106d-cb6d-44e9-bef7-b7720fa3a156 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:02.317607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d3106d-cb6d-44e9-bef7-b7720fa3a156 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:02.319162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d3106d-cb6d-44e9-bef7-b7720fa3a156 None admin] RBAC: Adding query filter params (role.id=ac68e1820c604122981fd5c4e2ebc447, scope.project.id=2e832f037cf845379232686bfc78f98d, user.id=d2b15600b8934082a524b413e42230b8) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:02.319802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d3106d-cb6d-44e9-bef7-b7720fa3a156 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:02.321882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d3106d-cb6d-44e9-bef7-b7720fa3a156 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:02.329822 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 113/224] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:02 2022] GET /identity/v3/role_assignments?user.id=d2b15600b8934082a524b413e42230b8&scope.project.id=2e832f037cf845379232686bfc78f98d&role.id=ac68e1820c604122981fd5c4e2ebc447 => generated 259 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:02.331936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d3106d-cb6d-44e9-bef7-b7720fa3a156 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:02.362098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8628c6c-5abb-4c59-8c27-7293196ec4f8 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:02.496962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d3106d-cb6d-44e9-bef7-b7720fa3a156 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:02.517547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfca7cd-6b25-4114-8db6-a184d75a9d7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:02.518687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfca7cd-6b25-4114-8db6-a184d75a9d7e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:02.519178 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfca7cd-6b25-4114-8db6-a184d75a9d7e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:02.519738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfca7cd-6b25-4114-8db6-a184d75a9d7e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:02.541462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:02.569309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfca7cd-6b25-4114-8db6-a184d75a9d7e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:02.575019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec901d68-0907-4490-911c-834597675585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:02.576147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec901d68-0907-4490-911c-834597675585 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-ebfca7cd-6b25-4114-8db6-a184d75a9d7e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:02.576437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:02.577306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:02.578099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:02.578099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:02.578099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:02.578099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:02.578099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:02.578099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:02.578099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:02.578099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:02.578099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:02.578099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:02.578099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:02.578099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:02.578713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec901d68-0907-4490-911c-834597675585 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:02.578713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec901d68-0907-4490-911c-834597675585 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:02.581430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 112/225] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:54:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 363 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:02.587763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfca7cd-6b25-4114-8db6-a184d75a9d7e None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:02.599156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec901d68-0907-4490-911c-834597675585 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-ec901d68-0907-4490-911c-834597675585 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:02.609937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:02.610932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:02.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:02.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:02.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:02.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:02.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:02.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:02.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:02.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:02.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:02.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:02.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:54:02.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:02.612902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 114/226] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:54:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:02.619959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:02.622712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec901d68-0907-4490-911c-834597675585 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:02.765208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfca7cd-6b25-4114-8db6-a184d75a9d7e None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:02.810364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec901d68-0907-4490-911c-834597675585 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:02.865672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f51e01-84be-4231-9596-d5f045fcddd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:02.866742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f51e01-84be-4231-9596-d5f045fcddd9 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:02.867268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f51e01-84be-4231-9596-d5f045fcddd9 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:02.867740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f51e01-84be-4231-9596-d5f045fcddd9 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:02.868954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f51e01-84be-4231-9596-d5f045fcddd9 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:02.869533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f51e01-84be-4231-9596-d5f045fcddd9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:02.871199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f51e01-84be-4231-9596-d5f045fcddd9 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:02.879624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb61858e-a9ba-4eab-b292-1698eb4ecf37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:02.879624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb61858e-a9ba-4eab-b292-1698eb4ecf37 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:02.879624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb61858e-a9ba-4eab-b292-1698eb4ecf37 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:02.879624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb61858e-a9ba-4eab-b292-1698eb4ecf37 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:02.880063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb61858e-a9ba-4eab-b292-1698eb4ecf37 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:02.880063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb61858e-a9ba-4eab-b292-1698eb4ecf37 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:02.880753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb61858e-a9ba-4eab-b292-1698eb4ecf37 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:02.896969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 115/227] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:02.898425 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 113/228] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:02.905135 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb61858e-a9ba-4eab-b292-1698eb4ecf37 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:02.912790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f51e01-84be-4231-9596-d5f045fcddd9 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:03.023947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:03.103926 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:03.156110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb61858e-a9ba-4eab-b292-1698eb4ecf37 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:03.168137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f51e01-84be-4231-9596-d5f045fcddd9 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:03.239011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3350331-4539-41e0-8039-b62480626890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:03.239823 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3350331-4539-41e0-8039-b62480626890 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:03.240275 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3350331-4539-41e0-8039-b62480626890 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:03.240558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3350331-4539-41e0-8039-b62480626890 None admin] PATH_INFO: `/v3/users/a1412db69e884863830ad4c27e6e9970` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:03.241936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfe0c2f-bf16-4393-ad5a-e25930bf8d1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:03.242112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3350331-4539-41e0-8039-b62480626890 None admin] RBAC: Authorizing `identity:get_user(user_id=a1412db69e884863830ad4c27e6e9970)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:03.242969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfe0c2f-bf16-4393-ad5a-e25930bf8d1b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:03.243495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfe0c2f-bf16-4393-ad5a-e25930bf8d1b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:03.243976 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfe0c2f-bf16-4393-ad5a-e25930bf8d1b None admin] PATH_INFO: `/v3/users/a1412db69e884863830ad4c27e6e9970` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:03.244401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3350331-4539-41e0-8039-b62480626890 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:03.245950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfe0c2f-bf16-4393-ad5a-e25930bf8d1b None admin] RBAC: Authorizing `identity:get_user(user_id=a1412db69e884863830ad4c27e6e9970)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:03.246445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 114/229] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:02 2022] GET /identity/v3/users/a1412db69e884863830ad4c27e6e9970 => generated 283 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:03.253695 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfe0c2f-bf16-4393-ad5a-e25930bf8d1b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:03.256953 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 116/230] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:02 2022] GET /identity/v3/users/a1412db69e884863830ad4c27e6e9970 => generated 283 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:03.260873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3350331-4539-41e0-8039-b62480626890 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:03.268662 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfe0c2f-bf16-4393-ad5a-e25930bf8d1b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:03.404713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfe0c2f-bf16-4393-ad5a-e25930bf8d1b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:03.428329 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:03.449567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e03488-5aef-46c4-a7ec-aa5ceabfcb22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:03.450521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e03488-5aef-46c4-a7ec-aa5ceabfcb22 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:03.450978 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e03488-5aef-46c4-a7ec-aa5ceabfcb22 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:03.451615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e03488-5aef-46c4-a7ec-aa5ceabfcb22 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:03.453038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e03488-5aef-46c4-a7ec-aa5ceabfcb22 None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:03.454855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e03488-5aef-46c4-a7ec-aa5ceabfcb22 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:03.458321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 117/231] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:03 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:03.476308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3350331-4539-41e0-8039-b62480626890 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:03.482784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e03488-5aef-46c4-a7ec-aa5ceabfcb22 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:03.546443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c31a8ba-0cee-4976-8c03-71be2981dd37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:03.546443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c31a8ba-0cee-4976-8c03-71be2981dd37 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:03.546443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c31a8ba-0cee-4976-8c03-71be2981dd37 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:03.546443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c31a8ba-0cee-4976-8c03-71be2981dd37 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:03.547344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c31a8ba-0cee-4976-8c03-71be2981dd37 None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:03.549939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c31a8ba-0cee-4976-8c03-71be2981dd37 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:03.559416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 115/232] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:03 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:03.565662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c31a8ba-0cee-4976-8c03-71be2981dd37 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:03.592363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:03.670910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e03488-5aef-46c4-a7ec-aa5ceabfcb22 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:03.741631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e186da7a-9aed-4aa9-a58d-edac61a81700 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:03.742829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e186da7a-9aed-4aa9-a58d-edac61a81700 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:03.742829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e186da7a-9aed-4aa9-a58d-edac61a81700 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:03.743110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e186da7a-9aed-4aa9-a58d-edac61a81700 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d/users/a1412db69e884863830ad4c27e6e9970/roles/0a2f5ca94533412088499eaccd7b2e21` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:03.755403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e186da7a-9aed-4aa9-a58d-edac61a81700 None admin] RBAC: Authorizing `identity:create_grant(project_id=2e832f037cf845379232686bfc78f98d, user_id=a1412db69e884863830ad4c27e6e9970, role_id=0a2f5ca94533412088499eaccd7b2e21)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:03.756986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e186da7a-9aed-4aa9-a58d-edac61a81700 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:03.782906 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 118/233] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:54:03 2022] PUT /identity/v3/projects/2e832f037cf845379232686bfc78f98d/users/a1412db69e884863830ad4c27e6e9970/roles/0a2f5ca94533412088499eaccd7b2e21 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:03.789570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c31a8ba-0cee-4976-8c03-71be2981dd37 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:03.856174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abee9cda-066e-4c3f-985d-6b119fccf43f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:03.856918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abee9cda-066e-4c3f-985d-6b119fccf43f None admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:03.857399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abee9cda-066e-4c3f-985d-6b119fccf43f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:03.857838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abee9cda-066e-4c3f-985d-6b119fccf43f None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d/users/a1412db69e884863830ad4c27e6e9970/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:03.860889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abee9cda-066e-4c3f-985d-6b119fccf43f None admin] RBAC: Authorizing `identity:create_grant(project_id=2e832f037cf845379232686bfc78f98d, user_id=a1412db69e884863830ad4c27e6e9970, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:03.863572 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abee9cda-066e-4c3f-985d-6b119fccf43f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:03.885834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 116/234] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:54:03 2022] PUT /identity/v3/projects/2e832f037cf845379232686bfc78f98d/users/a1412db69e884863830ad4c27e6e9970/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:04.422804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59160d7f-7974-44b5-b162-45e76aaf164b None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:04.423912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59160d7f-7974-44b5-b162-45e76aaf164b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:04.424990 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59160d7f-7974-44b5-b162-45e76aaf164b None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:04.427284 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 119/235] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:04.434779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c58ffb-5705-4ccb-8efe-2e21c1a6b455 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:04.435481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c58ffb-5705-4ccb-8efe-2e21c1a6b455 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:04.436111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c58ffb-5705-4ccb-8efe-2e21c1a6b455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:04.460721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f1c58ffb-5705-4ccb-8efe-2e21c1a6b455 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:04.462736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f1c58ffb-5705-4ccb-8efe-2e21c1a6b455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:04.557858 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 117/236] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:04.601270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59160d7f-7974-44b5-b162-45e76aaf164b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:04.638413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab6d5b1-f824-4d51-a310-f3314b93c7df None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:04.638692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab6d5b1-f824-4d51-a310-f3314b93c7df None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:04.639014 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab6d5b1-f824-4d51-a310-f3314b93c7df None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:04.640381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 118/237] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:04.646294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5916984-6824-4093-a774-5ef228e4a5f3 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:04.646888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5916984-6824-4093-a774-5ef228e4a5f3 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:04.647385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5916984-6824-4093-a774-5ef228e4a5f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:04.654244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:04.671579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-d5916984-6824-4093-a774-5ef228e4a5f3 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:04.673472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:04.673916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d5916984-6824-4093-a774-5ef228e4a5f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:04.769673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59160d7f-7974-44b5-b162-45e76aaf164b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:04.819546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 119/238] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:04.838965 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b3b184-d23b-428f-8ba0-83e911d6f859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:04.839646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b3b184-d23b-428f-8ba0-83e911d6f859 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:04.839923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b3b184-d23b-428f-8ba0-83e911d6f859 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:04.840192 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b3b184-d23b-428f-8ba0-83e911d6f859 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:04.846995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5916984-6824-4093-a774-5ef228e4a5f3 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:04.862660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b3b184-d23b-428f-8ba0-83e911d6f859 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-63b3b184-d23b-428f-8ba0-83e911d6f859 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:04.873364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:04.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:04.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:04.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:04.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:04.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:04.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:04.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:04.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:04.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:04.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:04.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:04.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:54:04.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:04.877931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 120/239] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:04 2022] GET /identity/v3/roles/reader => generated 84 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:04.884418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b3b184-d23b-428f-8ba0-83e911d6f859 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:05.066888 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b3b184-d23b-428f-8ba0-83e911d6f859 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:05.074720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:05.101737 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5916984-6824-4093-a774-5ef228e4a5f3 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:05.146341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7fbe34-597a-4eca-a7c4-5e93c009e43c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:05.148311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:05.148461 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7fbe34-597a-4eca-a7c4-5e93c009e43c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:05.149468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7fbe34-597a-4eca-a7c4-5e93c009e43c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:05.150030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7fbe34-597a-4eca-a7c4-5e93c009e43c None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:05.152137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7fbe34-597a-4eca-a7c4-5e93c009e43c None admin] RBAC: Adding query filter params (name=reader) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:05.152495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7fbe34-597a-4eca-a7c4-5e93c009e43c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:05.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7fbe34-597a-4eca-a7c4-5e93c009e43c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:05.156714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdea5bd-b94d-48d1-bda3-70f84fe92756 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:05.157601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdea5bd-b94d-48d1-bda3-70f84fe92756 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:05.158083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdea5bd-b94d-48d1-bda3-70f84fe92756 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:05.158658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdea5bd-b94d-48d1-bda3-70f84fe92756 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:05.160610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdea5bd-b94d-48d1-bda3-70f84fe92756 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:05.162306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdea5bd-b94d-48d1-bda3-70f84fe92756 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:05.164671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 120/240] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:05.166128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd9bb51-d2c2-425e-b15e-8d3ea7298ca6 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:05.166620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd9bb51-d2c2-425e-b15e-8d3ea7298ca6 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:05.167113 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd9bb51-d2c2-425e-b15e-8d3ea7298ca6 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:05.168684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 121/241] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:05.173871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd9bb51-d2c2-425e-b15e-8d3ea7298ca6 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:05.183389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 121/242] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:04 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:05.188159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7fbe34-597a-4eca-a7c4-5e93c009e43c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:05.331402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd9bb51-d2c2-425e-b15e-8d3ea7298ca6 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:05.350878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7fbe34-597a-4eca-a7c4-5e93c009e43c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:05.372074 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3951231a-e203-46f1-ba2c-ac76dc124a85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:05.373097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3951231a-e203-46f1-ba2c-ac76dc124a85 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:05.373621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3951231a-e203-46f1-ba2c-ac76dc124a85 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:05.374123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3951231a-e203-46f1-ba2c-ac76dc124a85 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:05.375265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3951231a-e203-46f1-ba2c-ac76dc124a85 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:05.377103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3951231a-e203-46f1-ba2c-ac76dc124a85 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:05.398146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc15e97-81d5-4211-bc9d-66130a8d1137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:05.398971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc15e97-81d5-4211-bc9d-66130a8d1137 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:05.399092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc15e97-81d5-4211-bc9d-66130a8d1137 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:05.399340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc15e97-81d5-4211-bc9d-66130a8d1137 None admin] PATH_INFO: `/v3/users/228c6d3fac794d6bbab189a29b8fab0e` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:05.406416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 122/243] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:54:05 2022] POST /identity/v3/users => generated 293 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:54:05.409843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8090c2c-5ddc-43f4-8055-37fe323dea63 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:05.410604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8090c2c-5ddc-43f4-8055-37fe323dea63 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:05.411195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8090c2c-5ddc-43f4-8055-37fe323dea63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:05.417351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc15e97-81d5-4211-bc9d-66130a8d1137 None admin] RBAC: Authorizing `identity:get_user(user_id=228c6d3fac794d6bbab189a29b8fab0e)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:05.419036 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc15e97-81d5-4211-bc9d-66130a8d1137 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:05.421152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 122/244] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:05 2022] GET /identity/v3/users/228c6d3fac794d6bbab189a29b8fab0e => generated 297 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:05.436479 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc15e97-81d5-4211-bc9d-66130a8d1137 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:05.450055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a8090c2c-5ddc-43f4-8055-37fe323dea63 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:05.451825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a8090c2c-5ddc-43f4-8055-37fe323dea63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:05.520778 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:05.547759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:05.584108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 123/245] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:05.615305 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8090c2c-5ddc-43f4-8055-37fe323dea63 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:05.696019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc15e97-81d5-4211-bc9d-66130a8d1137 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:05.756856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8090c2c-5ddc-43f4-8055-37fe323dea63 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:05.758087 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489a0829-77c2-42e8-8deb-7534146e0651 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:05.758778 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489a0829-77c2-42e8-8deb-7534146e0651 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:05.759086 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489a0829-77c2-42e8-8deb-7534146e0651 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:05.759411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489a0829-77c2-42e8-8deb-7534146e0651 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:05.800102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0550d82d-dd77-4c41-9f90-332c13905e15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:05.800825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0550d82d-dd77-4c41-9f90-332c13905e15 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:05.802147 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0550d82d-dd77-4c41-9f90-332c13905e15 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:05.802626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0550d82d-dd77-4c41-9f90-332c13905e15 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:05.822899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0550d82d-dd77-4c41-9f90-332c13905e15 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-0550d82d-dd77-4c41-9f90-332c13905e15 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:05.830420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:05.831526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:05.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 124/246] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:05.842156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0550d82d-dd77-4c41-9f90-332c13905e15 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:05.871387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489a0829-77c2-42e8-8deb-7534146e0651 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:05.871387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489a0829-77c2-42e8-8deb-7534146e0651 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:05.911261 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:05.962436 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:06.035674 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 123/247] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 600 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:06.042863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489a0829-77c2-42e8-8deb-7534146e0651 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:06.049882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0550d82d-dd77-4c41-9f90-332c13905e15 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:06.134554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d41a980-3b11-4014-9311-2131001bf348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:06.136182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d41a980-3b11-4014-9311-2131001bf348 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:06.137037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d41a980-3b11-4014-9311-2131001bf348 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:06.137725 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d41a980-3b11-4014-9311-2131001bf348 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:06.141072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d41a980-3b11-4014-9311-2131001bf348 None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:06.142162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d41a980-3b11-4014-9311-2131001bf348 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:06.146158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d41a980-3b11-4014-9311-2131001bf348 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:06.173740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 125/248] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:06.181091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489a0829-77c2-42e8-8deb-7534146e0651 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:06.185117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d41a980-3b11-4014-9311-2131001bf348 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:06.215862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:06.238810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45361260-7a90-4a90-a1da-63432e7e9d0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:06.240593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45361260-7a90-4a90-a1da-63432e7e9d0c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:06.240593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45361260-7a90-4a90-a1da-63432e7e9d0c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:06.240593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45361260-7a90-4a90-a1da-63432e7e9d0c None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:06.242012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45361260-7a90-4a90-a1da-63432e7e9d0c None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:06.243603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45361260-7a90-4a90-a1da-63432e7e9d0c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:06.247215 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 124/249] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:06 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:06.253540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45361260-7a90-4a90-a1da-63432e7e9d0c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:06.379384 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d41a980-3b11-4014-9311-2131001bf348 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:06.391863 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:06.397070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45361260-7a90-4a90-a1da-63432e7e9d0c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:06.449484 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70ecc06-42fc-4f22-971b-a36964612d10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:06.450617 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70ecc06-42fc-4f22-971b-a36964612d10 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:06.451149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70ecc06-42fc-4f22-971b-a36964612d10 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:06.451651 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70ecc06-42fc-4f22-971b-a36964612d10 None admin] PATH_INFO: `/v3/users/d2b15600b8934082a524b413e42230b8` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:06.453238 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70ecc06-42fc-4f22-971b-a36964612d10 None admin] RBAC: Authorizing `identity:get_user(user_id=d2b15600b8934082a524b413e42230b8)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:06.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70ecc06-42fc-4f22-971b-a36964612d10 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:06.457941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 126/250] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:06 2022] GET /identity/v3/users/d2b15600b8934082a524b413e42230b8 => generated 297 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:06.466835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70ecc06-42fc-4f22-971b-a36964612d10 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:06.489763 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9aec637-0629-4b7a-9159-2a93774dd463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:06.490299 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aec637-0629-4b7a-9159-2a93774dd463 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:06.490525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aec637-0629-4b7a-9159-2a93774dd463 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:06.490713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aec637-0629-4b7a-9159-2a93774dd463 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:06.491562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9aec637-0629-4b7a-9159-2a93774dd463 None admin] RBAC: Adding query filter params (role.id=de6529a9de0b48b2910599a3a56124f3, scope.project.id=2e832f037cf845379232686bfc78f98d, user.id=228c6d3fac794d6bbab189a29b8fab0e) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:06.491874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9aec637-0629-4b7a-9159-2a93774dd463 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:06.493346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9aec637-0629-4b7a-9159-2a93774dd463 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:06.512589 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 125/251] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:06 2022] GET /identity/v3/role_assignments?user.id=228c6d3fac794d6bbab189a29b8fab0e&scope.project.id=2e832f037cf845379232686bfc78f98d&role.id=de6529a9de0b48b2910599a3a56124f3 => generated 259 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:06.584154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f01076-922e-4e94-9ccc-e5cc3ab3e18c None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:06.584472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f01076-922e-4e94-9ccc-e5cc3ab3e18c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:06.584714 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f01076-922e-4e94-9ccc-e5cc3ab3e18c None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:06.586403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 126/252] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:06.591502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4f1126-15e7-4fc8-85aa-220a6e1b53fe None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:06.591810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4f1126-15e7-4fc8-85aa-220a6e1b53fe None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:06.592174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4f1126-15e7-4fc8-85aa-220a6e1b53fe None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:06.593760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 127/253] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:06.598381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07299969-d917-415e-bcfd-fe41226b725f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:06.598698 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07299969-d917-415e-bcfd-fe41226b725f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:06.598910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07299969-d917-415e-bcfd-fe41226b725f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:06.640325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-07299969-d917-415e-bcfd-fe41226b725f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:06.641569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-07299969-d917-415e-bcfd-fe41226b725f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:06.657819 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70ecc06-42fc-4f22-971b-a36964612d10 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:06.713783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9011429-0241-4a27-9537-e477b12154f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:06.714477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9011429-0241-4a27-9537-e477b12154f0 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:06.714742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9011429-0241-4a27-9537-e477b12154f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:06.715013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9011429-0241-4a27-9537-e477b12154f0 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:06.728423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9011429-0241-4a27-9537-e477b12154f0 None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:06.729950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9011429-0241-4a27-9537-e477b12154f0 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:06.731816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 127/254] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:06 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:06.733076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06390b2-80b1-4421-8394-83803bbd6300 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:06.733337 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06390b2-80b1-4421-8394-83803bbd6300 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:06.733609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06390b2-80b1-4421-8394-83803bbd6300 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:06.778583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-e06390b2-80b1-4421-8394-83803bbd6300 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:06.778583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e06390b2-80b1-4421-8394-83803bbd6300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:06.806765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:06.864820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 128/255] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:06.866155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07299969-d917-415e-bcfd-fe41226b725f None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:06.923912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:06.975638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 128/256] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:06.976846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06390b2-80b1-4421-8394-83803bbd6300 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:07.119835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07299969-d917-415e-bcfd-fe41226b725f None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:07.255933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06390b2-80b1-4421-8394-83803bbd6300 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:07.272750 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d81a55-6533-472f-9305-604d0bae5383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:07.273443 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d81a55-6533-472f-9305-604d0bae5383 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:07.273724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d81a55-6533-472f-9305-604d0bae5383 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:07.273999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d81a55-6533-472f-9305-604d0bae5383 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d/users/d2b15600b8934082a524b413e42230b8/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:07.277904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d81a55-6533-472f-9305-604d0bae5383 None admin] RBAC: Authorizing `identity:create_grant(project_id=2e832f037cf845379232686bfc78f98d, user_id=d2b15600b8934082a524b413e42230b8, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:07.279383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d81a55-6533-472f-9305-604d0bae5383 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:07.304493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 129/257] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:54:06 2022] PUT /identity/v3/projects/2e832f037cf845379232686bfc78f98d/users/d2b15600b8934082a524b413e42230b8/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 439 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:07.305126 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d81a55-6533-472f-9305-604d0bae5383 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:07.309704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33883bf8-e5cc-41cb-84a6-3c639cd249a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:07.310360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33883bf8-e5cc-41cb-84a6-3c639cd249a7 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:07.310647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33883bf8-e5cc-41cb-84a6-3c639cd249a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:07.310918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33883bf8-e5cc-41cb-84a6-3c639cd249a7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:07.339563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33883bf8-e5cc-41cb-84a6-3c639cd249a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-33883bf8-e5cc-41cb-84a6-3c639cd249a7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:07.344915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:07.345804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:07.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:07.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:07.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:07.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:07.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:07.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:07.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:07.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:07.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:07.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:07.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:54:07.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:07.349751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 129/258] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:54:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 374 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:07.355791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33883bf8-e5cc-41cb-84a6-3c639cd249a7 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:07.363613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:07.451757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:07.469332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d81a55-6533-472f-9305-604d0bae5383 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:07.547733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33883bf8-e5cc-41cb-84a6-3c639cd249a7 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:07.553121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de47831-9c13-4681-8fd4-c257e438cb3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:07.554565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de47831-9c13-4681-8fd4-c257e438cb3e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:07.555094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de47831-9c13-4681-8fd4-c257e438cb3e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:07.555619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de47831-9c13-4681-8fd4-c257e438cb3e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:07.581205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de47831-9c13-4681-8fd4-c257e438cb3e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-1de47831-9c13-4681-8fd4-c257e438cb3e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:07.593490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:07.594456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:07.595223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 130/259] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:54:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:07.613053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de47831-9c13-4681-8fd4-c257e438cb3e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:07.620503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb60f94a-497e-42a3-bfee-44282b1b770f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:07.621157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb60f94a-497e-42a3-bfee-44282b1b770f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:07.621652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb60f94a-497e-42a3-bfee-44282b1b770f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:07.621860 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb60f94a-497e-42a3-bfee-44282b1b770f None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:07.623366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb60f94a-497e-42a3-bfee-44282b1b770f None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:07.623366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb60f94a-497e-42a3-bfee-44282b1b770f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:07.624469 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb60f94a-497e-42a3-bfee-44282b1b770f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:07.642251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 130/260] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:07.647863 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb60f94a-497e-42a3-bfee-44282b1b770f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:07.811883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:07.817517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de47831-9c13-4681-8fd4-c257e438cb3e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:07.820798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb60f94a-497e-42a3-bfee-44282b1b770f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:07.830949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:07.859660 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a45e29b-5f57-465c-9010-661bc7a28391 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:07.860302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a45e29b-5f57-465c-9010-661bc7a28391 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:07.860579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a45e29b-5f57-465c-9010-661bc7a28391 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:07.860852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a45e29b-5f57-465c-9010-661bc7a28391 None admin] PATH_INFO: `/v3/users/a1412db69e884863830ad4c27e6e9970` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:07.862316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a45e29b-5f57-465c-9010-661bc7a28391 None admin] RBAC: Authorizing `identity:get_user(user_id=a1412db69e884863830ad4c27e6e9970)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:07.863983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a45e29b-5f57-465c-9010-661bc7a28391 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:07.866013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 131/261] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:07 2022] GET /identity/v3/users/a1412db69e884863830ad4c27e6e9970 => generated 283 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:07.871087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a45e29b-5f57-465c-9010-661bc7a28391 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:07.901913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6785d5b7-bb07-4cad-8702-e02dff994519 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:07.902545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6785d5b7-bb07-4cad-8702-e02dff994519 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:07.902817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6785d5b7-bb07-4cad-8702-e02dff994519 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:07.903084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6785d5b7-bb07-4cad-8702-e02dff994519 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:07.903837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6785d5b7-bb07-4cad-8702-e02dff994519 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:07.904139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6785d5b7-bb07-4cad-8702-e02dff994519 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:07.905545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6785d5b7-bb07-4cad-8702-e02dff994519 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:07.931021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 131/262] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:07.936553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6785d5b7-bb07-4cad-8702-e02dff994519 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:08.000692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a45e29b-5f57-465c-9010-661bc7a28391 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:08.038940 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d8f5e7-77ae-4795-bdf0-b6fbbc1de2ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:08.039583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d8f5e7-77ae-4795-bdf0-b6fbbc1de2ea None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:08.039851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d8f5e7-77ae-4795-bdf0-b6fbbc1de2ea None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:08.040124 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d8f5e7-77ae-4795-bdf0-b6fbbc1de2ea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:08.074458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:08.115433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d8f5e7-77ae-4795-bdf0-b6fbbc1de2ea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:08.116908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d8f5e7-77ae-4795-bdf0-b6fbbc1de2ea None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:08.197552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6785d5b7-bb07-4cad-8702-e02dff994519 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:08.230491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 132/263] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 31 00:54:08.235999 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d8f5e7-77ae-4795-bdf0-b6fbbc1de2ea None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:08.273447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714c7add-176d-462f-991e-75a3fac79da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:08.274103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714c7add-176d-462f-991e-75a3fac79da5 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:08.274399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714c7add-176d-462f-991e-75a3fac79da5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:08.274659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714c7add-176d-462f-991e-75a3fac79da5 None admin] PATH_INFO: `/v3/users/a1412db69e884863830ad4c27e6e9970` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:08.280090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714c7add-176d-462f-991e-75a3fac79da5 None admin] RBAC: Authorizing `identity:get_user(user_id=a1412db69e884863830ad4c27e6e9970)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:08.281579 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714c7add-176d-462f-991e-75a3fac79da5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:08.283250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 132/264] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:07 2022] GET /identity/v3/users/a1412db69e884863830ad4c27e6e9970 => generated 283 bytes in 347 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:08.288907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714c7add-176d-462f-991e-75a3fac79da5 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:08.300511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:08.342164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d8f5e7-77ae-4795-bdf0-b6fbbc1de2ea None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:08.378297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a4ec17-d1b0-4538-8ac9-d92b4aca7797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:08.378937 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a4ec17-d1b0-4538-8ac9-d92b4aca7797 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:08.379214 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a4ec17-d1b0-4538-8ac9-d92b4aca7797 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:08.379513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a4ec17-d1b0-4538-8ac9-d92b4aca7797 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:08.380779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a4ec17-d1b0-4538-8ac9-d92b4aca7797 None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:08.381123 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:08.383072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a4ec17-d1b0-4538-8ac9-d92b4aca7797 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:08.384942 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 133/265] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:08 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:08.391888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a4ec17-d1b0-4538-8ac9-d92b4aca7797 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:08.503380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714c7add-176d-462f-991e-75a3fac79da5 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:08.529012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a4ec17-d1b0-4538-8ac9-d92b4aca7797 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:08.539786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dd5295-3c8b-454a-9755-356795ac1419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:08.540839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dd5295-3c8b-454a-9755-356795ac1419 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:08.541213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dd5295-3c8b-454a-9755-356795ac1419 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:08.541641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dd5295-3c8b-454a-9755-356795ac1419 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:08.587783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:08.595102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3aec68-695b-4506-974a-43da998d96bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:08.595718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3aec68-695b-4506-974a-43da998d96bf None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:08.596093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3aec68-695b-4506-974a-43da998d96bf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:08.596513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3aec68-695b-4506-974a-43da998d96bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:08.597313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3aec68-695b-4506-974a-43da998d96bf None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=2e832f037cf845379232686bfc78f98d, user.id=a1412db69e884863830ad4c27e6e9970) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:08.597625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3aec68-695b-4506-974a-43da998d96bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:08.599066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3aec68-695b-4506-974a-43da998d96bf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:08.610362 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 134/266] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:08 2022] GET /identity/v3/role_assignments?user.id=a1412db69e884863830ad4c27e6e9970&scope.project.id=2e832f037cf845379232686bfc78f98d&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 608 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:08.641866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60dd5295-3c8b-454a-9755-356795ac1419 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:08.643499 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60dd5295-3c8b-454a-9755-356795ac1419 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:08.688019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:08.780351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 133/267] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 492 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:08.785924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3aec68-695b-4506-974a-43da998d96bf None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:08.815371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23145b26-07d6-44db-9653-7d213282afb4 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:08.815697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23145b26-07d6-44db-9653-7d213282afb4 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:08.816029 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23145b26-07d6-44db-9653-7d213282afb4 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:08.817418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 134/268] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:54:08.823066 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa383cb-5604-45bc-8dd1-a7e689d27a87 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:08.823346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa383cb-5604-45bc-8dd1-a7e689d27a87 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:08.823639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa383cb-5604-45bc-8dd1-a7e689d27a87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:08.843345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-1aa383cb-5604-45bc-8dd1-a7e689d27a87 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:08.844696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1aa383cb-5604-45bc-8dd1-a7e689d27a87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:08.923915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 135/269] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:08.924942 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3aec68-695b-4506-974a-43da998d96bf None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:08.954437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa383cb-5604-45bc-8dd1-a7e689d27a87 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:08.959868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556b9f3b-c682-49dd-b854-dee32588314e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:08.960511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556b9f3b-c682-49dd-b854-dee32588314e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:08.960969 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556b9f3b-c682-49dd-b854-dee32588314e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:08.961242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556b9f3b-c682-49dd-b854-dee32588314e None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:08.962426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556b9f3b-c682-49dd-b854-dee32588314e None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:08.964024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556b9f3b-c682-49dd-b854-dee32588314e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:08.966000 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 135/270] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:08 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:08.975681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556b9f3b-c682-49dd-b854-dee32588314e None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:09.040688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:09.041769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:09.082608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa383cb-5604-45bc-8dd1-a7e689d27a87 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:09.087224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556b9f3b-c682-49dd-b854-dee32588314e None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:09.123239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22153b64-6787-4c45-887a-2dd217ace783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:09.124114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22153b64-6787-4c45-887a-2dd217ace783 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:09.124388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22153b64-6787-4c45-887a-2dd217ace783 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:09.124682 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22153b64-6787-4c45-887a-2dd217ace783 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:09.126174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22153b64-6787-4c45-887a-2dd217ace783 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:09.128056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22153b64-6787-4c45-887a-2dd217ace783 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:09.130675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 136/271] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:09.137186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22153b64-6787-4c45-887a-2dd217ace783 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:09.152794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02db505-a7d8-4631-964c-8a06294dde27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:09.153433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02db505-a7d8-4631-964c-8a06294dde27 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:09.153711 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02db505-a7d8-4631-964c-8a06294dde27 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:09.154021 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02db505-a7d8-4631-964c-8a06294dde27 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:09.154827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02db505-a7d8-4631-964c-8a06294dde27 None admin] RBAC: Adding query filter params (role.id=0a2f5ca94533412088499eaccd7b2e21, scope.project.id=2e832f037cf845379232686bfc78f98d, user.id=a1412db69e884863830ad4c27e6e9970) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:09.155131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02db505-a7d8-4631-964c-8a06294dde27 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:09.156667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02db505-a7d8-4631-964c-8a06294dde27 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:09.168320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 136/272] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:08 2022] GET /identity/v3/role_assignments?user.id=a1412db69e884863830ad4c27e6e9970&scope.project.id=2e832f037cf845379232686bfc78f98d&role.id=0a2f5ca94533412088499eaccd7b2e21 => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:09.272331 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22153b64-6787-4c45-887a-2dd217ace783 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:09.312294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6d8f62-08fe-4606-900c-938fcd61bf8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:09.312924 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6d8f62-08fe-4606-900c-938fcd61bf8a None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:09.313204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6d8f62-08fe-4606-900c-938fcd61bf8a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:09.313480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6d8f62-08fe-4606-900c-938fcd61bf8a None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:09.314181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6d8f62-08fe-4606-900c-938fcd61bf8a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:09.315774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6d8f62-08fe-4606-900c-938fcd61bf8a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:09.324378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:09.342459 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 137/273] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:54:09 2022] POST /identity/v3/users => generated 293 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 00:54:09.362682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9c73a6-3c79-42f3-a3e4-35c7ea390789 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:09.363023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9c73a6-3c79-42f3-a3e4-35c7ea390789 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:09.363230 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9c73a6-3c79-42f3-a3e4-35c7ea390789 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:09.369323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 137/274] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:09 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:09.374968 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cab8205-7b5f-4d44-abb6-1706c93bec99 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:09.375272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cab8205-7b5f-4d44-abb6-1706c93bec99 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:09.375663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cab8205-7b5f-4d44-abb6-1706c93bec99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:09.403430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6cab8205-7b5f-4d44-abb6-1706c93bec99 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:09.404706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6cab8205-7b5f-4d44-abb6-1706c93bec99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:09.492694 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 138/275] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:09.515586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9c73a6-3c79-42f3-a3e4-35c7ea390789 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:09.690623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9c73a6-3c79-42f3-a3e4-35c7ea390789 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:09.719789 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:09.729224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3b3b48-4ce1-4f74-9dd4-18ba88e924d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:09.730010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3b3b48-4ce1-4f74-9dd4-18ba88e924d1 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:09.730354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3b3b48-4ce1-4f74-9dd4-18ba88e924d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:09.730626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3b3b48-4ce1-4f74-9dd4-18ba88e924d1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:09.743815 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3b3b48-4ce1-4f74-9dd4-18ba88e924d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-8f3b3b48-4ce1-4f74-9dd4-18ba88e924d1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:09.749353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:09.750244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:09.751102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 138/276] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:09.758476 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cab8205-7b5f-4d44-abb6-1706c93bec99 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:09.813433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cf29ae-3402-446b-a8d9-46b5655b7d50 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:09.813978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cf29ae-3402-446b-a8d9-46b5655b7d50 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:09.814394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cf29ae-3402-446b-a8d9-46b5655b7d50 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:09.816718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 139/277] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:09 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:09.822183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f1d8b-c410-4870-ac90-1ac10176416f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:09.822803 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f1d8b-c410-4870-ac90-1ac10176416f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:09.823439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f1d8b-c410-4870-ac90-1ac10176416f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:09.850320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-961f1d8b-c410-4870-ac90-1ac10176416f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:09.851954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-961f1d8b-c410-4870-ac90-1ac10176416f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:09.885475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cab8205-7b5f-4d44-abb6-1706c93bec99 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:09.913464 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:09.932154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4201a1-71f5-4932-855a-01fdb8c68a14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:09.933069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4201a1-71f5-4932-855a-01fdb8c68a14 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:09.933355 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4201a1-71f5-4932-855a-01fdb8c68a14 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:09.933629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4201a1-71f5-4932-855a-01fdb8c68a14 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:09.934720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4201a1-71f5-4932-855a-01fdb8c68a14 None admin] RBAC: Adding query filter params (name=member) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:09.935033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4201a1-71f5-4932-855a-01fdb8c68a14 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:09.936929 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4201a1-71f5-4932-855a-01fdb8c68a14 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:09.945705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 140/278] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:09.949504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 139/279] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:09.955835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961f1d8b-c410-4870-ac90-1ac10176416f None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:09.978161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4201a1-71f5-4932-855a-01fdb8c68a14 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:10.140503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4201a1-71f5-4932-855a-01fdb8c68a14 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:10.207828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453dd512-31d6-46de-b335-eb83a6c7a958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:10.208665 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453dd512-31d6-46de-b335-eb83a6c7a958 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:10.209067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453dd512-31d6-46de-b335-eb83a6c7a958 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:10.209565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453dd512-31d6-46de-b335-eb83a6c7a958 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:10.213500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961f1d8b-c410-4870-ac90-1ac10176416f None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:10.227862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453dd512-31d6-46de-b335-eb83a6c7a958 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-453dd512-31d6-46de-b335-eb83a6c7a958 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:10.234733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:10.235795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:10.237245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 140/280] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:09 2022] GET /identity/v3/roles/reader => generated 84 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:10.243823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453dd512-31d6-46de-b335-eb83a6c7a958 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:10.266092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:10.301720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:10.333246 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c057a4c8-1c89-4f1f-8b47-09502519047e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:10.334264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c057a4c8-1c89-4f1f-8b47-09502519047e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:10.334686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c057a4c8-1c89-4f1f-8b47-09502519047e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:10.335097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c057a4c8-1c89-4f1f-8b47-09502519047e None admin] PATH_INFO: `/v3/users/9dfc0aab9a204461ab4b9e22e1124a79` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:10.355553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c057a4c8-1c89-4f1f-8b47-09502519047e None admin] RBAC: Authorizing `identity:get_user(user_id=9dfc0aab9a204461ab4b9e22e1124a79)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:10.357433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c057a4c8-1c89-4f1f-8b47-09502519047e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:10.359591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 141/281] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:09 2022] GET /identity/v3/users/9dfc0aab9a204461ab4b9e22e1124a79 => generated 293 bytes in 405 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:10.366339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c057a4c8-1c89-4f1f-8b47-09502519047e None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:10.366492 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453dd512-31d6-46de-b335-eb83a6c7a958 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:10.451812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a8d1bc-1cfa-4176-84f7-81611fcd0da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:10.453377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a8d1bc-1cfa-4176-84f7-81611fcd0da5 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:10.453871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a8d1bc-1cfa-4176-84f7-81611fcd0da5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:10.454784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a8d1bc-1cfa-4176-84f7-81611fcd0da5 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:10.456550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a8d1bc-1cfa-4176-84f7-81611fcd0da5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:10.456997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a8d1bc-1cfa-4176-84f7-81611fcd0da5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:10.458710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a8d1bc-1cfa-4176-84f7-81611fcd0da5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:10.476102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 141/282] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:10 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:10.476102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3f51e6-e626-4032-ac76-fc492bd532e5 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:10.476102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3f51e6-e626-4032-ac76-fc492bd532e5 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:10.476390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3f51e6-e626-4032-ac76-fc492bd532e5 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:10.479427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 142/283] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:10.480484 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3f51e6-e626-4032-ac76-fc492bd532e5 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:10.545765 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c057a4c8-1c89-4f1f-8b47-09502519047e None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:10.593247 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a69f90-8f1c-4733-b8ec-7b3a121a855a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:10.593959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a69f90-8f1c-4733-b8ec-7b3a121a855a None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:10.594137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a69f90-8f1c-4733-b8ec-7b3a121a855a None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:10.594393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a69f90-8f1c-4733-b8ec-7b3a121a855a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:10.595185 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a69f90-8f1c-4733-b8ec-7b3a121a855a None admin] RBAC: Adding query filter params (role.id=ac68e1820c604122981fd5c4e2ebc447, user.id=9dfc0aab9a204461ab4b9e22e1124a79, scope.system=all) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:10.596812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a69f90-8f1c-4733-b8ec-7b3a121a855a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:10.597616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a69f90-8f1c-4733-b8ec-7b3a121a855a None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:10.603677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 142/284] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Jul 31 00:54:10 2022] GET /identity/v3/role_assignments?user.id=9dfc0aab9a204461ab4b9e22e1124a79&scope.system=all&role.id=ac68e1820c604122981fd5c4e2ebc447 => generated 226 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:10.605054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8c336d-4664-4a86-b6ed-763ecedd802f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:10.605340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8c336d-4664-4a86-b6ed-763ecedd802f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:10.605610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8c336d-4664-4a86-b6ed-763ecedd802f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:10.637019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3f51e6-e626-4032-ac76-fc492bd532e5 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:10.640448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:10.647051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-5d8c336d-4664-4a86-b6ed-763ecedd802f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:10.648359 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5d8c336d-4664-4a86-b6ed-763ecedd802f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:10.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af35020d-48f4-4574-8792-8ea1c326b5ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:10.690211 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af35020d-48f4-4574-8792-8ea1c326b5ee None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:10.690707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af35020d-48f4-4574-8792-8ea1c326b5ee None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:10.691202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af35020d-48f4-4574-8792-8ea1c326b5ee None admin] PATH_INFO: `/v3/users/228c6d3fac794d6bbab189a29b8fab0e` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:10.692637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af35020d-48f4-4574-8792-8ea1c326b5ee None admin] RBAC: Authorizing `identity:get_user(user_id=228c6d3fac794d6bbab189a29b8fab0e)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:10.694948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af35020d-48f4-4574-8792-8ea1c326b5ee None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:10.696884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 143/285] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:10 2022] GET /identity/v3/users/228c6d3fac794d6bbab189a29b8fab0e => generated 297 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:10.708204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af35020d-48f4-4574-8792-8ea1c326b5ee None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:10.738218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:10.815131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 143/286] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:10.840064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8c336d-4664-4a86-b6ed-763ecedd802f None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:10.914370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af35020d-48f4-4574-8792-8ea1c326b5ee None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:10.996081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29baf363-9f19-4405-baec-df06616f16e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:10.996742 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29baf363-9f19-4405-baec-df06616f16e7 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:10.997022 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29baf363-9f19-4405-baec-df06616f16e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:10.997394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29baf363-9f19-4405-baec-df06616f16e7 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:11.000161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29baf363-9f19-4405-baec-df06616f16e7 None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:11.001824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29baf363-9f19-4405-baec-df06616f16e7 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:11.003644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 144/287] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:10 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:11.010671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29baf363-9f19-4405-baec-df06616f16e7 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:11.025513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8c336d-4664-4a86-b6ed-763ecedd802f None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:11.101283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:11.104084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eb97a1-ef02-45fd-a9d4-cfd5830be0f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:11.104747 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eb97a1-ef02-45fd-a9d4-cfd5830be0f0 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:11.105028 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eb97a1-ef02-45fd-a9d4-cfd5830be0f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:11.105295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eb97a1-ef02-45fd-a9d4-cfd5830be0f0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:11.133140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05eb97a1-ef02-45fd-a9d4-cfd5830be0f0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-05eb97a1-ef02-45fd-a9d4-cfd5830be0f0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:11.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:11.143669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:11.144616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 144/288] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:11.155796 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eb97a1-ef02-45fd-a9d4-cfd5830be0f0 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:11.224615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:11.231638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29baf363-9f19-4405-baec-df06616f16e7 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:11.300761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26beb0cf-edd2-4477-ac40-954955230054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:11.302514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26beb0cf-edd2-4477-ac40-954955230054 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:11.302514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26beb0cf-edd2-4477-ac40-954955230054 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:11.302708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26beb0cf-edd2-4477-ac40-954955230054 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d/users/228c6d3fac794d6bbab189a29b8fab0e/roles/de6529a9de0b48b2910599a3a56124f3` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:11.308784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26beb0cf-edd2-4477-ac40-954955230054 None admin] RBAC: Authorizing `identity:create_grant(project_id=2e832f037cf845379232686bfc78f98d, user_id=228c6d3fac794d6bbab189a29b8fab0e, role_id=de6529a9de0b48b2910599a3a56124f3)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:11.310252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26beb0cf-edd2-4477-ac40-954955230054 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:11.329852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eb97a1-ef02-45fd-a9d4-cfd5830be0f0 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:11.343382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 145/289] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:54:11 2022] PUT /identity/v3/projects/2e832f037cf845379232686bfc78f98d/users/228c6d3fac794d6bbab189a29b8fab0e/roles/de6529a9de0b48b2910599a3a56124f3 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:11.385325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81122dc8-017a-4b61-a372-775909169871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:11.386418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81122dc8-017a-4b61-a372-775909169871 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:11.386898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81122dc8-017a-4b61-a372-775909169871 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:11.387387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81122dc8-017a-4b61-a372-775909169871 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:11.388368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81122dc8-017a-4b61-a372-775909169871 None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:11.388899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81122dc8-017a-4b61-a372-775909169871 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:11.390581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81122dc8-017a-4b61-a372-775909169871 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:11.406143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 145/290] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:11.413149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26beb0cf-edd2-4477-ac40-954955230054 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:11.610383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:11.613862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26beb0cf-edd2-4477-ac40-954955230054 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:11.662437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2dae681-b809-4224-924c-71b1fb50d626 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:11.663344 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dae681-b809-4224-924c-71b1fb50d626 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:11.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dae681-b809-4224-924c-71b1fb50d626 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:11.664216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dae681-b809-4224-924c-71b1fb50d626 None admin] PATH_INFO: `/v3/users/d2b15600b8934082a524b413e42230b8` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:11.665605 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2dae681-b809-4224-924c-71b1fb50d626 None admin] RBAC: Authorizing `identity:get_user(user_id=d2b15600b8934082a524b413e42230b8)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:11.667326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2dae681-b809-4224-924c-71b1fb50d626 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:11.670444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 146/291] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:11 2022] GET /identity/v3/users/d2b15600b8934082a524b413e42230b8 => generated 297 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:11.677066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81122dc8-017a-4b61-a372-775909169871 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:11.830818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:11.841624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81122dc8-017a-4b61-a372-775909169871 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:11.898919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d059fc-2daf-4ccb-9d05-5ff764f33a3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:11.899491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d059fc-2daf-4ccb-9d05-5ff764f33a3a None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:11.899772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d059fc-2daf-4ccb-9d05-5ff764f33a3a None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:11.900043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d059fc-2daf-4ccb-9d05-5ff764f33a3a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:11.987084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a088d31-84a3-40fb-85b2-a59c89e5ea23 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:11.987651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a088d31-84a3-40fb-85b2-a59c89e5ea23 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:11.987727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a088d31-84a3-40fb-85b2-a59c89e5ea23 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:11.989261 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 147/292] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:54:11.995982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1019ef-ebe7-4b6a-bf1c-9010386c947c None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:11.996252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1019ef-ebe7-4b6a-bf1c-9010386c947c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:11.996528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1019ef-ebe7-4b6a-bf1c-9010386c947c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:12.014915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d059fc-2daf-4ccb-9d05-5ff764f33a3a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:12.016407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d059fc-2daf-4ccb-9d05-5ff764f33a3a None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:12.018194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6d1019ef-ebe7-4b6a-bf1c-9010386c947c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:12.019667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6d1019ef-ebe7-4b6a-bf1c-9010386c947c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:12.105296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 148/293] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:12.131646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1019ef-ebe7-4b6a-bf1c-9010386c947c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:12.176246 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:12.191055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 146/294] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 511 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 31 00:54:12.191055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffccd95-6d53-4ed9-9560-b610a3b826c5 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:12.191055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffccd95-6d53-4ed9-9560-b610a3b826c5 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:12.191055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffccd95-6d53-4ed9-9560-b610a3b826c5 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:12.191055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 147/295] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:12.197006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffccd95-6d53-4ed9-9560-b610a3b826c5 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:12.259706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:12.264487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1019ef-ebe7-4b6a-bf1c-9010386c947c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:12.307128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf863aa-86a1-480d-94ec-4d8f540dbad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:12.308106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf863aa-86a1-480d-94ec-4d8f540dbad0 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:12.308536 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf863aa-86a1-480d-94ec-4d8f540dbad0 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:12.308975 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf863aa-86a1-480d-94ec-4d8f540dbad0 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:12.321076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffccd95-6d53-4ed9-9560-b610a3b826c5 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:12.322638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf863aa-86a1-480d-94ec-4d8f540dbad0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-8cf863aa-86a1-480d-94ec-4d8f540dbad0 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:12.328338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:12.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:12.330045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:12.330045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:12.330045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:12.330045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:12.330045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:12.330045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:12.330045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:12.330045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:12.330045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:12.330045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:12.330045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:54:12.330045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:12.333110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 149/296] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:12 2022] GET /identity/v3/roles/reader => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:12.334961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926b2af7-9fbf-4fc1-9f43-41600d7ae586 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:12.335439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926b2af7-9fbf-4fc1-9f43-41600d7ae586 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:12.335888 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926b2af7-9fbf-4fc1-9f43-41600d7ae586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:12.357838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-926b2af7-9fbf-4fc1-9f43-41600d7ae586 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:12.359539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-926b2af7-9fbf-4fc1-9f43-41600d7ae586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:12.365855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f68753-1ad1-4a8c-a64f-98ca27f74acd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:12.366649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f68753-1ad1-4a8c-a64f-98ca27f74acd None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:12.367031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f68753-1ad1-4a8c-a64f-98ca27f74acd None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:12.367447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f68753-1ad1-4a8c-a64f-98ca27f74acd None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:12.368780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f68753-1ad1-4a8c-a64f-98ca27f74acd None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:12.370621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f68753-1ad1-4a8c-a64f-98ca27f74acd None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:12.372326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 148/297] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:12 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:12.377825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f68753-1ad1-4a8c-a64f-98ca27f74acd None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:12.445587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:12.449068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 150/298] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:12.450296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926b2af7-9fbf-4fc1-9f43-41600d7ae586 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:12.514487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f68753-1ad1-4a8c-a64f-98ca27f74acd None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:12.554330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d94cd17-0dca-430a-87e9-893fc83eadb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:12.554952 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d94cd17-0dca-430a-87e9-893fc83eadb3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:12.555230 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d94cd17-0dca-430a-87e9-893fc83eadb3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:12.555546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d94cd17-0dca-430a-87e9-893fc83eadb3 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:12.556500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d94cd17-0dca-430a-87e9-893fc83eadb3 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:12.556609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d94cd17-0dca-430a-87e9-893fc83eadb3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:12.557931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d94cd17-0dca-430a-87e9-893fc83eadb3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:12.567438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 149/299] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:12 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:12.568275 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d94cd17-0dca-430a-87e9-893fc83eadb3 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:12.573042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:12.587829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926b2af7-9fbf-4fc1-9f43-41600d7ae586 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:12.625611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed8d283-a16a-4d7f-8136-ea8507662c84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:12.626576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed8d283-a16a-4d7f-8136-ea8507662c84 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:12.627611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed8d283-a16a-4d7f-8136-ea8507662c84 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:12.628201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed8d283-a16a-4d7f-8136-ea8507662c84 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:12.629337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed8d283-a16a-4d7f-8136-ea8507662c84 None admin] RBAC: Adding query filter params (role.id=ac68e1820c604122981fd5c4e2ebc447, scope.project.id=2e832f037cf845379232686bfc78f98d, user.id=d2b15600b8934082a524b413e42230b8) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:12.629848 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed8d283-a16a-4d7f-8136-ea8507662c84 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:12.631679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed8d283-a16a-4d7f-8136-ea8507662c84 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:12.647578 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 151/300] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:12 2022] GET /identity/v3/role_assignments?user.id=d2b15600b8934082a524b413e42230b8&scope.project.id=2e832f037cf845379232686bfc78f98d&role.id=ac68e1820c604122981fd5c4e2ebc447 => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:12.648937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed8d283-a16a-4d7f-8136-ea8507662c84 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:12.678177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d94cd17-0dca-430a-87e9-893fc83eadb3 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:12.713531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:12.722859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a710e8-869b-477e-850a-0548169aea8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:12.723553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a710e8-869b-477e-850a-0548169aea8e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:12.723833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a710e8-869b-477e-850a-0548169aea8e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:12.724098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a710e8-869b-477e-850a-0548169aea8e None admin] PATH_INFO: `/v3/domains/default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:12.726072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a710e8-869b-477e-850a-0548169aea8e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:12.727590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a710e8-869b-477e-850a-0548169aea8e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:12.729508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 150/301] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:12.735055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a710e8-869b-477e-850a-0548169aea8e None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:12.891160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed8d283-a16a-4d7f-8136-ea8507662c84 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:12.910332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a710e8-869b-477e-850a-0548169aea8e None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:12.928867 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6f3503-a0a8-45ea-8083-1991e1c6d5cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:12.929869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6f3503-a0a8-45ea-8083-1991e1c6d5cd None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:12.930438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6f3503-a0a8-45ea-8083-1991e1c6d5cd None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:12.930940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6f3503-a0a8-45ea-8083-1991e1c6d5cd None admin] PATH_INFO: `/v3/users/710dfffab72c42f8b9d4ba5ce7c24e3c` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:12.946315 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6f3503-a0a8-45ea-8083-1991e1c6d5cd None admin] RBAC: Authorizing `identity:get_user(user_id=710dfffab72c42f8b9d4ba5ce7c24e3c)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:12.948348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6f3503-a0a8-45ea-8083-1991e1c6d5cd None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:12.949241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae55249-f4bf-4245-90ce-bbcc5548a741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:12.949942 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae55249-f4bf-4245-90ce-bbcc5548a741 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:12.950292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae55249-f4bf-4245-90ce-bbcc5548a741 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:12.950638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae55249-f4bf-4245-90ce-bbcc5548a741 None admin] PATH_INFO: `/v3/groups` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:12.951269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 152/302] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:12 2022] GET /identity/v3/users/710dfffab72c42f8b9d4ba5ce7c24e3c => generated 293 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:12.951498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae55249-f4bf-4245-90ce-bbcc5548a741 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:12.952967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae55249-f4bf-4245-90ce-bbcc5548a741 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:12.957542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6f3503-a0a8-45ea-8083-1991e1c6d5cd None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:12.972093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:12.988220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 151/303] 10.222.0.42 () {60 vars in 1134 bytes} [Sun Jul 31 00:54:12 2022] POST /identity/v3/groups => generated 229 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 00:54:13.094945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6f3503-a0a8-45ea-8083-1991e1c6d5cd None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:13.140590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec924c2-f10f-4e4b-8753-00d8bfd78bcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:13.141663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec924c2-f10f-4e4b-8753-00d8bfd78bcc None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:13.142185 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec924c2-f10f-4e4b-8753-00d8bfd78bcc None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:13.142683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec924c2-f10f-4e4b-8753-00d8bfd78bcc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:13.143730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec924c2-f10f-4e4b-8753-00d8bfd78bcc None admin] RBAC: Adding query filter params (role.id=de6529a9de0b48b2910599a3a56124f3, user.id=710dfffab72c42f8b9d4ba5ce7c24e3c, scope.system=all) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:13.144269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec924c2-f10f-4e4b-8753-00d8bfd78bcc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:13.146028 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec924c2-f10f-4e4b-8753-00d8bfd78bcc None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:13.155275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 153/304] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Jul 31 00:54:12 2022] GET /identity/v3/role_assignments?user.id=710dfffab72c42f8b9d4ba5ce7c24e3c&scope.system=all&role.id=de6529a9de0b48b2910599a3a56124f3 => generated 226 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:13.335568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559daa20-20c2-46cf-a727-052e9aba4f99 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:13.336055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559daa20-20c2-46cf-a727-052e9aba4f99 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:13.336055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559daa20-20c2-46cf-a727-052e9aba4f99 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:13.337649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 152/305] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:13.344866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f853dbf-0085-4ba7-9a13-9e09ce21debe None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:13.345290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f853dbf-0085-4ba7-9a13-9e09ce21debe None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:13.346209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f853dbf-0085-4ba7-9a13-9e09ce21debe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:13.365025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-8f853dbf-0085-4ba7-9a13-9e09ce21debe None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:13.366216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8f853dbf-0085-4ba7-9a13-9e09ce21debe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:13.438683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 154/306] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:13.460357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559daa20-20c2-46cf-a727-052e9aba4f99 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:13.514990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:13.570475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559daa20-20c2-46cf-a727-052e9aba4f99 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:13.607567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6193e1c-3da4-48f7-8639-64267e3b520e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:13.608130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6193e1c-3da4-48f7-8639-64267e3b520e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:13.608407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6193e1c-3da4-48f7-8639-64267e3b520e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:13.608681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6193e1c-3da4-48f7-8639-64267e3b520e None admin] PATH_INFO: `/v3/roles/member` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:13.619992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6193e1c-3da4-48f7-8639-64267e3b520e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-b6193e1c-3da4-48f7-8639-64267e3b520e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:13.625418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:13.626291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:13.627523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 153/307] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:13.636466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f853dbf-0085-4ba7-9a13-9e09ce21debe None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:13.640667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:13.659649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f54dae-f0cd-4f9b-aa44-cd858ba49d75 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:13.659919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f54dae-f0cd-4f9b-aa44-cd858ba49d75 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:13.660204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f54dae-f0cd-4f9b-aa44-cd858ba49d75 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:13.661818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 154/308] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:13.666631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9ae828-ca15-4d31-8c77-f57810253b36 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:13.666916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9ae828-ca15-4d31-8c77-f57810253b36 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:13.667534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9ae828-ca15-4d31-8c77-f57810253b36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:13.685900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-0b9ae828-ca15-4d31-8c77-f57810253b36 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:13.687098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0b9ae828-ca15-4d31-8c77-f57810253b36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:13.761130 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f853dbf-0085-4ba7-9a13-9e09ce21debe None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:13.770529 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 155/309] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:54:13.793947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9ae828-ca15-4d31-8c77-f57810253b36 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:13.802480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c052bf1e-3f51-46e7-8e54-c9f3b1ac369e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:13.803388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c052bf1e-3f51-46e7-8e54-c9f3b1ac369e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:13.803847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c052bf1e-3f51-46e7-8e54-c9f3b1ac369e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:13.804407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c052bf1e-3f51-46e7-8e54-c9f3b1ac369e None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:13.805296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c052bf1e-3f51-46e7-8e54-c9f3b1ac369e None admin] RBAC: Adding query filter params (name=member) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:13.805773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c052bf1e-3f51-46e7-8e54-c9f3b1ac369e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:13.807323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c052bf1e-3f51-46e7-8e54-c9f3b1ac369e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:13.816853 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 155/310] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:13.822667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c052bf1e-3f51-46e7-8e54-c9f3b1ac369e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:13.839264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:13.894515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:13.903794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9ae828-ca15-4d31-8c77-f57810253b36 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:13.938349 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c052bf1e-3f51-46e7-8e54-c9f3b1ac369e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:13.941615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdfc570-939e-4285-af4e-47936f80355c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:13.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdfc570-939e-4285-af4e-47936f80355c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:13.942148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdfc570-939e-4285-af4e-47936f80355c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:13.942422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdfc570-939e-4285-af4e-47936f80355c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:13.953969 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdfc570-939e-4285-af4e-47936f80355c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-dcdfc570-939e-4285-af4e-47936f80355c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:13.959243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:13.960155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:13.960928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 156/311] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:13 2022] GET /identity/v3/roles/reader => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:13.966321 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdfc570-939e-4285-af4e-47936f80355c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:13.979452 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a34e96b-ac08-4e83-94c4-98e64ae50cea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:13.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a34e96b-ac08-4e83-94c4-98e64ae50cea None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:13.980851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a34e96b-ac08-4e83-94c4-98e64ae50cea None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:13.981297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a34e96b-ac08-4e83-94c4-98e64ae50cea None admin] PATH_INFO: `/v3/users/9dfc0aab9a204461ab4b9e22e1124a79` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:13.982692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a34e96b-ac08-4e83-94c4-98e64ae50cea None admin] RBAC: Authorizing `identity:get_user(user_id=9dfc0aab9a204461ab4b9e22e1124a79)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:13.984352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a34e96b-ac08-4e83-94c4-98e64ae50cea None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:13.986238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 156/312] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:13 2022] GET /identity/v3/users/9dfc0aab9a204461ab4b9e22e1124a79 => generated 293 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:13.991905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a34e96b-ac08-4e83-94c4-98e64ae50cea None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:14.092740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:14.097776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdfc570-939e-4285-af4e-47936f80355c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:14.099781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a34e96b-ac08-4e83-94c4-98e64ae50cea None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:14.140583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f023075-31f0-4299-8292-ab1c01febd00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:14.141640 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f023075-31f0-4299-8292-ab1c01febd00 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:14.142141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f023075-31f0-4299-8292-ab1c01febd00 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:14.142654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f023075-31f0-4299-8292-ab1c01febd00 None admin] PATH_INFO: `/v3/system/users/9dfc0aab9a204461ab4b9e22e1124a79/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:14.144468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f62636-2b18-4573-8c19-56f5f9aac0c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:14.144871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f023075-31f0-4299-8292-ab1c01febd00 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=9dfc0aab9a204461ab4b9e22e1124a79, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:14.145487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f62636-2b18-4573-8c19-56f5f9aac0c4 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:14.145913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f62636-2b18-4573-8c19-56f5f9aac0c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:14.146328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f62636-2b18-4573-8c19-56f5f9aac0c4 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:14.146439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f023075-31f0-4299-8292-ab1c01febd00 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:14.147278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f62636-2b18-4573-8c19-56f5f9aac0c4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:14.147764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f62636-2b18-4573-8c19-56f5f9aac0c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:14.149304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f62636-2b18-4573-8c19-56f5f9aac0c4 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:14.150691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:14.159657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 157/314] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:13 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:14.164621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 157/314] 10.222.0.42 () {58 vars in 1334 bytes} [Sun Jul 31 00:54:13 2022] PUT /identity/v3/system/users/9dfc0aab9a204461ab4b9e22e1124a79/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 00:54:14.165949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f62636-2b18-4573-8c19-56f5f9aac0c4 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:14.287670 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f62636-2b18-4573-8c19-56f5f9aac0c4 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:14.326670 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5991845-28c1-4bfe-bbd9-22f4958e24bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:14.327345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5991845-28c1-4bfe-bbd9-22f4958e24bf None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:14.327659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5991845-28c1-4bfe-bbd9-22f4958e24bf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:14.327933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5991845-28c1-4bfe-bbd9-22f4958e24bf None admin] PATH_INFO: `/v3/users/228c6d3fac794d6bbab189a29b8fab0e` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:14.329308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5991845-28c1-4bfe-bbd9-22f4958e24bf None admin] RBAC: Authorizing `identity:get_user(user_id=228c6d3fac794d6bbab189a29b8fab0e)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:14.331415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5991845-28c1-4bfe-bbd9-22f4958e24bf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:14.333396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 158/315] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:14 2022] GET /identity/v3/users/228c6d3fac794d6bbab189a29b8fab0e => generated 297 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:14.339735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f023075-31f0-4299-8292-ab1c01febd00 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:14.447857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f023075-31f0-4299-8292-ab1c01febd00 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:14.484835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce4607d-93c8-4854-a9c9-1233fd1e535e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:14.485466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce4607d-93c8-4854-a9c9-1233fd1e535e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:14.485745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce4607d-93c8-4854-a9c9-1233fd1e535e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:14.486020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce4607d-93c8-4854-a9c9-1233fd1e535e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:14.559941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:14.568664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce4607d-93c8-4854-a9c9-1233fd1e535e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:14.570402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce4607d-93c8-4854-a9c9-1233fd1e535e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:14.679415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 158/316] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:14.684595 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5991845-28c1-4bfe-bbd9-22f4958e24bf None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:14.721027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:14.807782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5991845-28c1-4bfe-bbd9-22f4958e24bf None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:14.845231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6de3fbb-8e80-4481-bc99-25f5dbbcea65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:14.846497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6de3fbb-8e80-4481-bc99-25f5dbbcea65 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:14.846824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6de3fbb-8e80-4481-bc99-25f5dbbcea65 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:14.847161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6de3fbb-8e80-4481-bc99-25f5dbbcea65 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:14.848483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6de3fbb-8e80-4481-bc99-25f5dbbcea65 None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:14.850006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6de3fbb-8e80-4481-bc99-25f5dbbcea65 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:14.852384 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 159/317] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:14 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:14.860524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce4607d-93c8-4854-a9c9-1233fd1e535e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:14.976638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce4607d-93c8-4854-a9c9-1233fd1e535e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:14.979655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:15.013787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57d086a-d8cb-42ad-922d-bf23f37c73d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:15.014416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57d086a-d8cb-42ad-922d-bf23f37c73d9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:15.014698 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57d086a-d8cb-42ad-922d-bf23f37c73d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:15.014973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57d086a-d8cb-42ad-922d-bf23f37c73d9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:15.015797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57d086a-d8cb-42ad-922d-bf23f37c73d9 None admin] RBAC: Adding query filter params (role.id=de6529a9de0b48b2910599a3a56124f3, scope.project.id=2e832f037cf845379232686bfc78f98d, user.id=228c6d3fac794d6bbab189a29b8fab0e) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:15.016110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57d086a-d8cb-42ad-922d-bf23f37c73d9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:15.017561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57d086a-d8cb-42ad-922d-bf23f37c73d9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:15.024393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 159/318] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:14 2022] GET /identity/v3/role_assignments?user.id=228c6d3fac794d6bbab189a29b8fab0e&scope.project.id=2e832f037cf845379232686bfc78f98d&role.id=de6529a9de0b48b2910599a3a56124f3 => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:15.229988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e78176-06c1-49d4-b506-c7b21e16792c None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:15.230648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e78176-06c1-49d4-b506-c7b21e16792c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:15.231162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e78176-06c1-49d4-b506-c7b21e16792c None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:15.233065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 160/319] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:15.238835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8eda53-27ac-4700-b6e8-f90269b9292f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:15.239891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8eda53-27ac-4700-b6e8-f90269b9292f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:15.240365 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8eda53-27ac-4700-b6e8-f90269b9292f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:15.259528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ff8eda53-27ac-4700-b6e8-f90269b9292f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:15.261045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ff8eda53-27ac-4700-b6e8-f90269b9292f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:15.346581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 160/320] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:15.365756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47120548-125c-472a-885b-c739536ce907 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:15.366060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47120548-125c-472a-885b-c739536ce907 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:15.366306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47120548-125c-472a-885b-c739536ce907 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:15.368088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 161/321] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:15.368654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8eda53-27ac-4700-b6e8-f90269b9292f None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:15.372952 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4865c0-34fe-44b3-97c5-65e04670ff9b None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:15.373233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4865c0-34fe-44b3-97c5-65e04670ff9b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:15.373503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4865c0-34fe-44b3-97c5-65e04670ff9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:15.391167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-de4865c0-34fe-44b3-97c5-65e04670ff9b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:15.392386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-de4865c0-34fe-44b3-97c5-65e04670ff9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:15.469806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:15.471489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 162/322] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:54:15.484604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8eda53-27ac-4700-b6e8-f90269b9292f None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:15.492549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4865c0-34fe-44b3-97c5-65e04670ff9b None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:15.524679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:15.526438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234b64f9-c8ae-4508-b582-e7d30526b81c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:15.527101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234b64f9-c8ae-4508-b582-e7d30526b81c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:15.527404 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234b64f9-c8ae-4508-b582-e7d30526b81c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:15.527881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234b64f9-c8ae-4508-b582-e7d30526b81c None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:15.530238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234b64f9-c8ae-4508-b582-e7d30526b81c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:15.531768 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234b64f9-c8ae-4508-b582-e7d30526b81c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:15.534262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 161/323] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:15.539716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234b64f9-c8ae-4508-b582-e7d30526b81c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:15.610174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4865c0-34fe-44b3-97c5-65e04670ff9b None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:15.646491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af77d997-34d0-4a60-bf90-eb5ae67a1a7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:15.647135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af77d997-34d0-4a60-bf90-eb5ae67a1a7c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:15.647441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af77d997-34d0-4a60-bf90-eb5ae67a1a7c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:15.647716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af77d997-34d0-4a60-bf90-eb5ae67a1a7c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:15.655108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234b64f9-c8ae-4508-b582-e7d30526b81c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:15.659334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af77d997-34d0-4a60-bf90-eb5ae67a1a7c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-af77d997-34d0-4a60-bf90-eb5ae67a1a7c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:15.664309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:15.665177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:15.665968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 163/324] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:15 2022] GET /identity/v3/roles/reader => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:15.671159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af77d997-34d0-4a60-bf90-eb5ae67a1a7c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:15.694986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7ada9c-ff89-4dfb-8c97-2921b7067fc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:15.695788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7ada9c-ff89-4dfb-8c97-2921b7067fc4 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:15.696072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7ada9c-ff89-4dfb-8c97-2921b7067fc4 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:15.696345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7ada9c-ff89-4dfb-8c97-2921b7067fc4 None admin] PATH_INFO: `/v3/groups` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:15.697002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7ada9c-ff89-4dfb-8c97-2921b7067fc4 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:15.698489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7ada9c-ff89-4dfb-8c97-2921b7067fc4 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:15.715720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 162/325] 10.222.0.42 () {60 vars in 1134 bytes} [Sun Jul 31 00:54:15 2022] POST /identity/v3/groups => generated 226 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 00:54:15.768513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:15.784723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af77d997-34d0-4a60-bf90-eb5ae67a1a7c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:15.825196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e214e777-f0d9-4dd9-be96-e55a8edeb0b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:15.825820 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214e777-f0d9-4dd9-be96-e55a8edeb0b2 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:15.826092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214e777-f0d9-4dd9-be96-e55a8edeb0b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:15.826361 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214e777-f0d9-4dd9-be96-e55a8edeb0b2 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:15.827079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e214e777-f0d9-4dd9-be96-e55a8edeb0b2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:15.827394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e214e777-f0d9-4dd9-be96-e55a8edeb0b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:15.828824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e214e777-f0d9-4dd9-be96-e55a8edeb0b2 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:15.838332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 164/326] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:15 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:15.843988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7ada9c-ff89-4dfb-8c97-2921b7067fc4 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:15.869161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:15.955652 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7ada9c-ff89-4dfb-8c97-2921b7067fc4 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:15.992358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b32bfe-e229-4c5c-9f6a-038072547cf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:15.993013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b32bfe-e229-4c5c-9f6a-038072547cf8 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:15.993287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b32bfe-e229-4c5c-9f6a-038072547cf8 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:15.993565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b32bfe-e229-4c5c-9f6a-038072547cf8 None admin] PATH_INFO: `/v3/users/710dfffab72c42f8b9d4ba5ce7c24e3c` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:15.994756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b32bfe-e229-4c5c-9f6a-038072547cf8 None admin] RBAC: Authorizing `identity:get_user(user_id=710dfffab72c42f8b9d4ba5ce7c24e3c)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:15.996282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b32bfe-e229-4c5c-9f6a-038072547cf8 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:15.997920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 163/327] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:15 2022] GET /identity/v3/users/710dfffab72c42f8b9d4ba5ce7c24e3c => generated 293 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:16.003792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e214e777-f0d9-4dd9-be96-e55a8edeb0b2 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:16.116103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e214e777-f0d9-4dd9-be96-e55a8edeb0b2 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:16.156672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250f93b4-aaf3-4b3d-93e2-2d5ecf25c0ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:16.157460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250f93b4-aaf3-4b3d-93e2-2d5ecf25c0ad None admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:16.157749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250f93b4-aaf3-4b3d-93e2-2d5ecf25c0ad None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:16.158025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250f93b4-aaf3-4b3d-93e2-2d5ecf25c0ad None admin] PATH_INFO: `/v3/system/users/710dfffab72c42f8b9d4ba5ce7c24e3c/roles/de6529a9de0b48b2910599a3a56124f3` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:16.160152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250f93b4-aaf3-4b3d-93e2-2d5ecf25c0ad None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=710dfffab72c42f8b9d4ba5ce7c24e3c, role_id=de6529a9de0b48b2910599a3a56124f3)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:16.161539 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250f93b4-aaf3-4b3d-93e2-2d5ecf25c0ad None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:16.173422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 165/328] 10.222.0.42 () {58 vars in 1334 bytes} [Sun Jul 31 00:54:16 2022] PUT /identity/v3/system/users/710dfffab72c42f8b9d4ba5ce7c24e3c/roles/de6529a9de0b48b2910599a3a56124f3 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 00:54:16.373736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69280134-bfb3-4d9a-9d07-d49f71f72585 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:16.374229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69280134-bfb3-4d9a-9d07-d49f71f72585 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:16.374229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69280134-bfb3-4d9a-9d07-d49f71f72585 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:16.375747 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 164/329] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:16.381954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a20da1b-eb3c-48dc-ade0-cdcc82177a35 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:16.382242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a20da1b-eb3c-48dc-ade0-cdcc82177a35 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:16.382516 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a20da1b-eb3c-48dc-ade0-cdcc82177a35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:16.405537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-9a20da1b-eb3c-48dc-ade0-cdcc82177a35 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:16.406744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9a20da1b-eb3c-48dc-ade0-cdcc82177a35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:16.417266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:16.495319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 166/330] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:16.519381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69280134-bfb3-4d9a-9d07-d49f71f72585 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:16.632444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:16.646834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69280134-bfb3-4d9a-9d07-d49f71f72585 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:16.684792 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c704ce-8129-4c97-b773-99a098188876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:16.685492 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c704ce-8129-4c97-b773-99a098188876 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:16.685779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c704ce-8129-4c97-b773-99a098188876 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:16.686102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c704ce-8129-4c97-b773-99a098188876 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:16.698004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c704ce-8129-4c97-b773-99a098188876 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-62c704ce-8129-4c97-b773-99a098188876 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:16.703611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:16.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:16.706282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 165/331] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:16.711273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a20da1b-eb3c-48dc-ade0-cdcc82177a35 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:16.823169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a20da1b-eb3c-48dc-ade0-cdcc82177a35 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:16.872367 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d506d60-6c0c-4e63-aa22-272ab4b8edb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:16.873338 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d506d60-6c0c-4e63-aa22-272ab4b8edb8 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:16.873789 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d506d60-6c0c-4e63-aa22-272ab4b8edb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:16.874230 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d506d60-6c0c-4e63-aa22-272ab4b8edb8 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:16.875475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d506d60-6c0c-4e63-aa22-272ab4b8edb8 None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:16.875962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d506d60-6c0c-4e63-aa22-272ab4b8edb8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:16.879105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d506d60-6c0c-4e63-aa22-272ab4b8edb8 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:16.894697 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 167/332] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:16.900213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c704ce-8129-4c97-b773-99a098188876 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:17.021889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c704ce-8129-4c97-b773-99a098188876 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:17.058931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a371d6-02f9-4583-9d50-0f7e7d72face None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:17.059617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a371d6-02f9-4583-9d50-0f7e7d72face None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:17.059900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a371d6-02f9-4583-9d50-0f7e7d72face None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:17.060178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a371d6-02f9-4583-9d50-0f7e7d72face None admin] PATH_INFO: `/v3/users/9dfc0aab9a204461ab4b9e22e1124a79` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:17.061388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a371d6-02f9-4583-9d50-0f7e7d72face None admin] RBAC: Authorizing `identity:get_user(user_id=9dfc0aab9a204461ab4b9e22e1124a79)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:17.062947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a371d6-02f9-4583-9d50-0f7e7d72face None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:17.065549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 166/333] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:16 2022] GET /identity/v3/users/9dfc0aab9a204461ab4b9e22e1124a79 => generated 293 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:17.071309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d506d60-6c0c-4e63-aa22-272ab4b8edb8 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:17.095140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:17.205844 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d506d60-6c0c-4e63-aa22-272ab4b8edb8 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:17.251163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968f7396-39d8-4ccf-a9d2-07c140d661f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:17.251949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968f7396-39d8-4ccf-a9d2-07c140d661f6 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:17.252235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968f7396-39d8-4ccf-a9d2-07c140d661f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:17.252549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968f7396-39d8-4ccf-a9d2-07c140d661f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:17.253615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968f7396-39d8-4ccf-a9d2-07c140d661f6 None admin] RBAC: Adding query filter params (role.id=ac68e1820c604122981fd5c4e2ebc447, user.id=9dfc0aab9a204461ab4b9e22e1124a79, scope.system=all) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:17.254100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968f7396-39d8-4ccf-a9d2-07c140d661f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:17.256048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968f7396-39d8-4ccf-a9d2-07c140d661f6 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:17.269581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 168/334] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Jul 31 00:54:17 2022] GET /identity/v3/role_assignments?user.id=9dfc0aab9a204461ab4b9e22e1124a79&scope.system=all&role.id=ac68e1820c604122981fd5c4e2ebc447 => generated 510 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:18.286541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801dc6cf-cac5-457a-82b8-e496d31ba307 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:18.287244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801dc6cf-cac5-457a-82b8-e496d31ba307 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:18.287244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801dc6cf-cac5-457a-82b8-e496d31ba307 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:18.289174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 167/335] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:18.295546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aad0b31-d1bf-4016-ab3e-2d52c7374e0c None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:18.296091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aad0b31-d1bf-4016-ab3e-2d52c7374e0c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:18.296535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aad0b31-d1bf-4016-ab3e-2d52c7374e0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:18.318949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-6aad0b31-d1bf-4016-ab3e-2d52c7374e0c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:18.320170 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6aad0b31-d1bf-4016-ab3e-2d52c7374e0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:18.389451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:18.399039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 169/336] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:54:18.423203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801dc6cf-cac5-457a-82b8-e496d31ba307 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:18.445200 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:18.497238 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a26c7c-0084-495a-ace9-37b4fa435890 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:18.497557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a26c7c-0084-495a-ace9-37b4fa435890 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:18.497770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a26c7c-0084-495a-ace9-37b4fa435890 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:18.499283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 170/337] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:18.539463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1534e40a-69a0-4fc2-8315-76d594527dac None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:18.539463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1534e40a-69a0-4fc2-8315-76d594527dac None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:18.539463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1534e40a-69a0-4fc2-8315-76d594527dac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:18.568080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801dc6cf-cac5-457a-82b8-e496d31ba307 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:18.572476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-1534e40a-69a0-4fc2-8315-76d594527dac None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:18.573746 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-1534e40a-69a0-4fc2-8315-76d594527dac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:18.645057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44c746e-fe9c-486a-b2af-660b956399db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:18.645719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44c746e-fe9c-486a-b2af-660b956399db None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:18.645999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44c746e-fe9c-486a-b2af-660b956399db None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:18.646270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44c746e-fe9c-486a-b2af-660b956399db None admin] PATH_INFO: `/v3/roles/member` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:18.669047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44c746e-fe9c-486a-b2af-660b956399db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:18.669664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 171/338] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-d44c746e-fe9c-486a-b2af-660b956399db None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:18.679129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:18.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:18.681267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 168/339] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:18.686355 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1534e40a-69a0-4fc2-8315-76d594527dac None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:18.748533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44c746e-fe9c-486a-b2af-660b956399db None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:18.807195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1534e40a-69a0-4fc2-8315-76d594527dac None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:18.835797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:18.845323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462adfd4-d128-4723-b233-7cd1ced0c17e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:18.846041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462adfd4-d128-4723-b233-7cd1ced0c17e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:18.846413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462adfd4-d128-4723-b233-7cd1ced0c17e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:18.846696 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462adfd4-d128-4723-b233-7cd1ced0c17e None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:18.847549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462adfd4-d128-4723-b233-7cd1ced0c17e None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:18.847854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462adfd4-d128-4723-b233-7cd1ced0c17e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:18.849300 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462adfd4-d128-4723-b233-7cd1ced0c17e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:18.858514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 172/340] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:18.864584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462adfd4-d128-4723-b233-7cd1ced0c17e None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:18.885773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:18.977391 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44c746e-fe9c-486a-b2af-660b956399db None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:18.993753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462adfd4-d128-4723-b233-7cd1ced0c17e None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:19.033220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2452be-c448-4a4d-bd0a-598e9b032ad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:19.034160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2452be-c448-4a4d-bd0a-598e9b032ad1 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:19.034604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2452be-c448-4a4d-bd0a-598e9b032ad1 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:19.035064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2452be-c448-4a4d-bd0a-598e9b032ad1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:19.096428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43809511-5d54-46df-bbae-a884f4c5dbc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:19.097018 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43809511-5d54-46df-bbae-a884f4c5dbc3 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:19.097300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43809511-5d54-46df-bbae-a884f4c5dbc3 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:19.097571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43809511-5d54-46df-bbae-a884f4c5dbc3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:19.098452 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:19.114518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2452be-c448-4a4d-bd0a-598e9b032ad1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:19.116295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2452be-c448-4a4d-bd0a-598e9b032ad1 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:19.148065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43809511-5d54-46df-bbae-a884f4c5dbc3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-43809511-5d54-46df-bbae-a884f4c5dbc3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:19.153513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:19.154614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:19.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 169/341] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:54:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 407 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:19.180102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43809511-5d54-46df-bbae-a884f4c5dbc3 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:19.238863 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 173/342] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:19.245663 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2452be-c448-4a4d-bd0a-598e9b032ad1 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:19.347852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:19.367954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2452be-c448-4a4d-bd0a-598e9b032ad1 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:19.423089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eed2015-f86b-48e3-947b-a85e41ab78a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:19.424356 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eed2015-f86b-48e3-947b-a85e41ab78a3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:19.424634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eed2015-f86b-48e3-947b-a85e41ab78a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:19.424904 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eed2015-f86b-48e3-947b-a85e41ab78a3 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:19.427749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eed2015-f86b-48e3-947b-a85e41ab78a3 None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:19.429898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eed2015-f86b-48e3-947b-a85e41ab78a3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:19.432117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 174/343] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:19 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:19.437887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eed2015-f86b-48e3-947b-a85e41ab78a3 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:19.494231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43809511-5d54-46df-bbae-a884f4c5dbc3 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:19.539278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cf7273-a88c-40d1-a619-68178cf059a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:19.540239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cf7273-a88c-40d1-a619-68178cf059a2 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:19.540519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cf7273-a88c-40d1-a619-68178cf059a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:19.540885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cf7273-a88c-40d1-a619-68178cf059a2 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:19.541619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cf7273-a88c-40d1-a619-68178cf059a2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:19.541945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cf7273-a88c-40d1-a619-68178cf059a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:19.548255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cf7273-a88c-40d1-a619-68178cf059a2 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:19.551788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:19.561526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 170/344] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 382 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:19.562857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4472b948-96b7-4d4e-979c-9c8bbb405f97 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:19.563142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4472b948-96b7-4d4e-979c-9c8bbb405f97 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:19.563531 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4472b948-96b7-4d4e-979c-9c8bbb405f97 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:19.565012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 171/345] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:19.570124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4472b948-96b7-4d4e-979c-9c8bbb405f97 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:19.579232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eed2015-f86b-48e3-947b-a85e41ab78a3 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:19.618192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ba3e5e-5428-4e08-84ac-59f5a8881939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:19.619182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ba3e5e-5428-4e08-84ac-59f5a8881939 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:19.619709 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ba3e5e-5428-4e08-84ac-59f5a8881939 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:19.620192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ba3e5e-5428-4e08-84ac-59f5a8881939 None admin] PATH_INFO: `/v3/groups/41df40ce995b4b4380a5d89505417606` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:19.627088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ba3e5e-5428-4e08-84ac-59f5a8881939 None admin] RBAC: Authorizing `identity:get_group(group_id=41df40ce995b4b4380a5d89505417606)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:19.628873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ba3e5e-5428-4e08-84ac-59f5a8881939 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:19.631161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 175/346] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:19 2022] GET /identity/v3/groups/41df40ce995b4b4380a5d89505417606 => generated 226 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:19.632672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4398f13-4add-442d-af9b-ded06e5b084d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:19.633160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4398f13-4add-442d-af9b-ded06e5b084d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:19.633655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4398f13-4add-442d-af9b-ded06e5b084d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:19.660282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a4398f13-4add-442d-af9b-ded06e5b084d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:19.661780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a4398f13-4add-442d-af9b-ded06e5b084d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:19.695752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:19.710152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4472b948-96b7-4d4e-979c-9c8bbb405f97 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:19.746453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67f9d97-87bf-40c0-9e2a-fbf77ead6c57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:19.747077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67f9d97-87bf-40c0-9e2a-fbf77ead6c57 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:19.747368 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67f9d97-87bf-40c0-9e2a-fbf77ead6c57 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:19.747656 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67f9d97-87bf-40c0-9e2a-fbf77ead6c57 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:19.780184 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 176/347] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:19.781482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4398f13-4add-442d-af9b-ded06e5b084d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:19.822819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67f9d97-87bf-40c0-9e2a-fbf77ead6c57 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:19.825278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67f9d97-87bf-40c0-9e2a-fbf77ead6c57 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:19.839734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:19.918353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4398f13-4add-442d-af9b-ded06e5b084d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:19.952275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 172/348] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 31 00:54:19.953540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3dd82f-dc28-4456-a055-77666f43b13c None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:19.953825 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3dd82f-dc28-4456-a055-77666f43b13c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:19.954099 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3dd82f-dc28-4456-a055-77666f43b13c None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:19.960133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 173/349] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:19 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:19.961063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3dd82f-dc28-4456-a055-77666f43b13c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:19.971869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c98763-e70a-4d9c-8799-f60efb1e875c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:19.973010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c98763-e70a-4d9c-8799-f60efb1e875c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:19.978701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c98763-e70a-4d9c-8799-f60efb1e875c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:19.979568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c98763-e70a-4d9c-8799-f60efb1e875c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:19.982592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c98763-e70a-4d9c-8799-f60efb1e875c None admin] RBAC: Adding query filter params (group.id=41df40ce995b4b4380a5d89505417606, role.id=ac68e1820c604122981fd5c4e2ebc447, scope.project.id=c0935b3a5a454c70bf75163763e11b34) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:19.983550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c98763-e70a-4d9c-8799-f60efb1e875c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:19.990003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c98763-e70a-4d9c-8799-f60efb1e875c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:20.004423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 177/350] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Jul 31 00:54:19 2022] GET /identity/v3/role_assignments?group.id=41df40ce995b4b4380a5d89505417606&scope.project.id=c0935b3a5a454c70bf75163763e11b34&role.id=ac68e1820c604122981fd5c4e2ebc447 => generated 260 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:20.007886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52edabc4-9a8a-4e7c-962f-d90083c0af9c None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:20.008797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52edabc4-9a8a-4e7c-962f-d90083c0af9c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:20.009549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52edabc4-9a8a-4e7c-962f-d90083c0af9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:20.042896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-52edabc4-9a8a-4e7c-962f-d90083c0af9c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:20.044335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-52edabc4-9a8a-4e7c-962f-d90083c0af9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:20.060237 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:20.102359 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:20.110941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3dd82f-dc28-4456-a055-77666f43b13c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:20.159189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be8efd9-6a58-403e-8da8-f913a585e902 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:20.160154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be8efd9-6a58-403e-8da8-f913a585e902 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:20.160730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be8efd9-6a58-403e-8da8-f913a585e902 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:20.161278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be8efd9-6a58-403e-8da8-f913a585e902 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:20.186569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be8efd9-6a58-403e-8da8-f913a585e902 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-9be8efd9-6a58-403e-8da8-f913a585e902 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:20.200505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:20.201509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:20.202338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 174/351] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:20.203041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be8efd9-6a58-403e-8da8-f913a585e902 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:20.205887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 178/352] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:20.212071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52edabc4-9a8a-4e7c-962f-d90083c0af9c None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:20.332746 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be8efd9-6a58-403e-8da8-f913a585e902 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:20.373872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe5be43-a9dc-420e-800f-aa704dc67d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:20.374786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe5be43-a9dc-420e-800f-aa704dc67d69 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:20.375311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe5be43-a9dc-420e-800f-aa704dc67d69 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:20.376324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe5be43-a9dc-420e-800f-aa704dc67d69 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:20.377816 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe5be43-a9dc-420e-800f-aa704dc67d69 None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:20.379745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe5be43-a9dc-420e-800f-aa704dc67d69 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:20.381770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 175/353] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:20 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:20.382751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe5be43-a9dc-420e-800f-aa704dc67d69 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:20.421997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:20.501170 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52edabc4-9a8a-4e7c-962f-d90083c0af9c None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:20.549421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b451581d-bab8-4e95-bef6-20dd85273d21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:20.549850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe5be43-a9dc-420e-800f-aa704dc67d69 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:20.551025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b451581d-bab8-4e95-bef6-20dd85273d21 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:20.551823 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b451581d-bab8-4e95-bef6-20dd85273d21 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:20.552324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b451581d-bab8-4e95-bef6-20dd85273d21 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:20.553307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b451581d-bab8-4e95-bef6-20dd85273d21 None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:20.553621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b451581d-bab8-4e95-bef6-20dd85273d21 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:20.555066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b451581d-bab8-4e95-bef6-20dd85273d21 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:20.564995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 179/354] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:20.565873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b451581d-bab8-4e95-bef6-20dd85273d21 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:20.577963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:20.644942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e32d254-8110-4d7c-acfa-9ab4b0f7fc18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:20.645665 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e32d254-8110-4d7c-acfa-9ab4b0f7fc18 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:20.645952 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e32d254-8110-4d7c-acfa-9ab4b0f7fc18 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:20.646227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e32d254-8110-4d7c-acfa-9ab4b0f7fc18 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:20.672239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e32d254-8110-4d7c-acfa-9ab4b0f7fc18 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-1e32d254-8110-4d7c-acfa-9ab4b0f7fc18 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:20.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:20.679142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:20.679948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 176/355] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:20 2022] GET /identity/v3/roles/reader => generated 84 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:20.682242 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e32d254-8110-4d7c-acfa-9ab4b0f7fc18 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:20.692447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b451581d-bab8-4e95-bef6-20dd85273d21 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:20.729930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515b5ddc-e3ff-4dff-a659-99a1a1f1a431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:20.735389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515b5ddc-e3ff-4dff-a659-99a1a1f1a431 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:20.735389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515b5ddc-e3ff-4dff-a659-99a1a1f1a431 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:20.735389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515b5ddc-e3ff-4dff-a659-99a1a1f1a431 None admin] PATH_INFO: `/v3/groups/41df40ce995b4b4380a5d89505417606` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:20.735389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515b5ddc-e3ff-4dff-a659-99a1a1f1a431 None admin] RBAC: Authorizing `identity:get_group(group_id=41df40ce995b4b4380a5d89505417606)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:20.735389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515b5ddc-e3ff-4dff-a659-99a1a1f1a431 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:20.735389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 180/356] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:20 2022] GET /identity/v3/groups/41df40ce995b4b4380a5d89505417606 => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:20.736200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515b5ddc-e3ff-4dff-a659-99a1a1f1a431 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:20.815830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:20.831277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:20.848750 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e32d254-8110-4d7c-acfa-9ab4b0f7fc18 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:20.855314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515b5ddc-e3ff-4dff-a659-99a1a1f1a431 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:20.888731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c1f6c2-c4a4-4de6-9643-8aa371e3b4ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:20.889751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c1f6c2-c4a4-4de6-9643-8aa371e3b4ad None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:20.890210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c1f6c2-c4a4-4de6-9643-8aa371e3b4ad None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:20.890693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c1f6c2-c4a4-4de6-9643-8aa371e3b4ad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:20.891630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c77cfa-39ca-4bf9-afdc-4d82c8c35b79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:20.893017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c77cfa-39ca-4bf9-afdc-4d82c8c35b79 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:20.893413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c77cfa-39ca-4bf9-afdc-4d82c8c35b79 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:20.893800 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c77cfa-39ca-4bf9-afdc-4d82c8c35b79 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:20.894627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c77cfa-39ca-4bf9-afdc-4d82c8c35b79 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:20.896041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c77cfa-39ca-4bf9-afdc-4d82c8c35b79 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:20.896566 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c77cfa-39ca-4bf9-afdc-4d82c8c35b79 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:20.913246 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 181/357] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:20 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:20.914098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c77cfa-39ca-4bf9-afdc-4d82c8c35b79 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:20.970959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c1f6c2-c4a4-4de6-9643-8aa371e3b4ad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:20.972480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c1f6c2-c4a4-4de6-9643-8aa371e3b4ad None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:21.052033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c77cfa-39ca-4bf9-afdc-4d82c8c35b79 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:21.096483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:21.101241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82150a08-a5ca-496f-abf8-f5b6f57902db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:21.102043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82150a08-a5ca-496f-abf8-f5b6f57902db None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:21.102343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82150a08-a5ca-496f-abf8-f5b6f57902db None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:21.102678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82150a08-a5ca-496f-abf8-f5b6f57902db None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:21.103619 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82150a08-a5ca-496f-abf8-f5b6f57902db None admin] RBAC: Adding query filter params (group.id=41df40ce995b4b4380a5d89505417606, role.id=0a2f5ca94533412088499eaccd7b2e21, scope.project.id=c0935b3a5a454c70bf75163763e11b34) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:21.103910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82150a08-a5ca-496f-abf8-f5b6f57902db None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:21.106291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82150a08-a5ca-496f-abf8-f5b6f57902db None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:21.109228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:21.116494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 182/358] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Jul 31 00:54:20 2022] GET /identity/v3/role_assignments?group.id=41df40ce995b4b4380a5d89505417606&scope.project.id=c0935b3a5a454c70bf75163763e11b34&role.id=0a2f5ca94533412088499eaccd7b2e21 => generated 260 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:21.117748 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7433629-85f7-428a-bdcf-9c8773b96e0a None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:21.118176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7433629-85f7-428a-bdcf-9c8773b96e0a None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:21.118926 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7433629-85f7-428a-bdcf-9c8773b96e0a None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:21.122256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 183/359] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:21 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:21.127376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd4c93c-aaba-4af8-b621-122d8ca651cb None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:21.127376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd4c93c-aaba-4af8-b621-122d8ca651cb None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:21.127376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd4c93c-aaba-4af8-b621-122d8ca651cb None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:21.127376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 184/360] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:21.127376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd4c93c-aaba-4af8-b621-122d8ca651cb None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:21.128375 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 177/361] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:21.132582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a71a3-ce78-47d6-aba6-f2a688c61bbd None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:21.133120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a71a3-ce78-47d6-aba6-f2a688c61bbd None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:21.133595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a71a3-ce78-47d6-aba6-f2a688c61bbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:21.154285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-2c6a71a3-ce78-47d6-aba6-f2a688c61bbd None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:21.155843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2c6a71a3-ce78-47d6-aba6-f2a688c61bbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:21.233305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 178/362] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:54:21.234590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6a71a3-ce78-47d6-aba6-f2a688c61bbd None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:21.259786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd4c93c-aaba-4af8-b621-122d8ca651cb None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:21.321203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eab69db-91d7-4bf8-9d57-26d6579ef01c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:21.321869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eab69db-91d7-4bf8-9d57-26d6579ef01c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:21.322148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eab69db-91d7-4bf8-9d57-26d6579ef01c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:21.322421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eab69db-91d7-4bf8-9d57-26d6579ef01c None admin] PATH_INFO: `/v3/users/710dfffab72c42f8b9d4ba5ce7c24e3c` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:21.324322 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eab69db-91d7-4bf8-9d57-26d6579ef01c None admin] RBAC: Authorizing `identity:get_user(user_id=710dfffab72c42f8b9d4ba5ce7c24e3c)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:21.325786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eab69db-91d7-4bf8-9d57-26d6579ef01c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:21.327650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 185/363] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:54:21 2022] GET /identity/v3/users/710dfffab72c42f8b9d4ba5ce7c24e3c => generated 293 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:21.330787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc337d07-0a3e-4c8b-950c-c1295d33f60e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:21.331958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc337d07-0a3e-4c8b-950c-c1295d33f60e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:21.331958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc337d07-0a3e-4c8b-950c-c1295d33f60e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:21.360496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-cc337d07-0a3e-4c8b-950c-c1295d33f60e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:21.362341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-cc337d07-0a3e-4c8b-950c-c1295d33f60e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:21.391708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6a71a3-ce78-47d6-aba6-f2a688c61bbd None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:21.420898 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:21.437683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57867480-c93f-4e15-845c-4afa428ede6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:21.438656 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57867480-c93f-4e15-845c-4afa428ede6b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:21.439119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57867480-c93f-4e15-845c-4afa428ede6b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:21.440556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57867480-c93f-4e15-845c-4afa428ede6b None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:21.442116 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57867480-c93f-4e15-845c-4afa428ede6b None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:21.444545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57867480-c93f-4e15-845c-4afa428ede6b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:21.446927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 179/364] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:21 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:21.448074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57867480-c93f-4e15-845c-4afa428ede6b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:21.448576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:21.463519 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 186/365] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:21.464446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc337d07-0a3e-4c8b-950c-c1295d33f60e None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:21.600151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57867480-c93f-4e15-845c-4afa428ede6b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:21.626970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc337d07-0a3e-4c8b-950c-c1295d33f60e None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:21.651757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99064e5-47b0-4189-9607-ae74a4eb68ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:21.652396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99064e5-47b0-4189-9607-ae74a4eb68ef None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:21.652671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99064e5-47b0-4189-9607-ae74a4eb68ef None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:21.652937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99064e5-47b0-4189-9607-ae74a4eb68ef None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:21.664880 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99064e5-47b0-4189-9607-ae74a4eb68ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-f99064e5-47b0-4189-9607-ae74a4eb68ef None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:21.670364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:21.671252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:21.672080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:21.672080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:21.672080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:21.672080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:21.672080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:21.672080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:21.672080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:21.672080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:21.672080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:21.672080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:21.672080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:54:21.672080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:21.675769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 180/366] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:54:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:21.676967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99064e5-47b0-4189-9607-ae74a4eb68ef None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:21.677911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42219b8f-402e-4aea-8783-5e28da1bc738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:21.678858 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42219b8f-402e-4aea-8783-5e28da1bc738 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:21.679339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42219b8f-402e-4aea-8783-5e28da1bc738 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:21.679862 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42219b8f-402e-4aea-8783-5e28da1bc738 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:21.680855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42219b8f-402e-4aea-8783-5e28da1bc738 None admin] RBAC: Adding query filter params (role.id=de6529a9de0b48b2910599a3a56124f3, user.id=710dfffab72c42f8b9d4ba5ce7c24e3c, scope.system=all) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:21.681376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42219b8f-402e-4aea-8783-5e28da1bc738 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:21.683393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42219b8f-402e-4aea-8783-5e28da1bc738 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:21.700297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 187/367] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Jul 31 00:54:21 2022] GET /identity/v3/role_assignments?user.id=710dfffab72c42f8b9d4ba5ce7c24e3c&scope.system=all&role.id=de6529a9de0b48b2910599a3a56124f3 => generated 510 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:21.701946 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42219b8f-402e-4aea-8783-5e28da1bc738 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:21.735496 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:21.781843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:21.818404 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99064e5-47b0-4189-9607-ae74a4eb68ef None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:21.836192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42219b8f-402e-4aea-8783-5e28da1bc738 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:21.869721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8790da6-39dc-4cd0-9999-dc5390fabacb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:21.870392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8790da6-39dc-4cd0-9999-dc5390fabacb None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:21.871719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8790da6-39dc-4cd0-9999-dc5390fabacb None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:21.871719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8790da6-39dc-4cd0-9999-dc5390fabacb None admin] PATH_INFO: `/v3/groups/41df40ce995b4b4380a5d89505417606` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:21.873367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8790da6-39dc-4cd0-9999-dc5390fabacb None admin] RBAC: Authorizing `identity:get_group(group_id=41df40ce995b4b4380a5d89505417606)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:21.875103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8790da6-39dc-4cd0-9999-dc5390fabacb None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:21.877155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 181/368] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:21 2022] GET /identity/v3/groups/41df40ce995b4b4380a5d89505417606 => generated 226 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:21.877867 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8790da6-39dc-4cd0-9999-dc5390fabacb None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:21.893204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c10eb5-448b-4946-9944-33b4f3cc6fed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:21.893865 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c10eb5-448b-4946-9944-33b4f3cc6fed None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:21.894297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c10eb5-448b-4946-9944-33b4f3cc6fed None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:21.894624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c10eb5-448b-4946-9944-33b4f3cc6fed None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:21.906690 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c10eb5-448b-4946-9944-33b4f3cc6fed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-a0c10eb5-448b-4946-9944-33b4f3cc6fed None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:21.913500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:21.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:21.916218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 188/369] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:54:21 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:21.919619 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c10eb5-448b-4946-9944-33b4f3cc6fed None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:22.014125 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8790da6-39dc-4cd0-9999-dc5390fabacb None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:22.031238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:22.056435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c10eb5-448b-4946-9944-33b4f3cc6fed None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:22.058721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99391c9b-c497-4002-bff8-402c22af2237 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:22.060178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99391c9b-c497-4002-bff8-402c22af2237 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:22.060178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99391c9b-c497-4002-bff8-402c22af2237 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:22.060178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99391c9b-c497-4002-bff8-402c22af2237 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:22.060720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99391c9b-c497-4002-bff8-402c22af2237 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:22.061020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99391c9b-c497-4002-bff8-402c22af2237 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:22.062414 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99391c9b-c497-4002-bff8-402c22af2237 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:22.072313 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 182/370] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:22.073207 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99391c9b-c497-4002-bff8-402c22af2237 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:22.078694 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:22.101854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a95db3-e6bc-43e2-b335-ca5ecb0a1681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:22.103119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a95db3-e6bc-43e2-b335-ca5ecb0a1681 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:22.103675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a95db3-e6bc-43e2-b335-ca5ecb0a1681 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:22.104158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a95db3-e6bc-43e2-b335-ca5ecb0a1681 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:22.105657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a95db3-e6bc-43e2-b335-ca5ecb0a1681 None admin] RBAC: Adding query filter params (group.id=41df40ce995b4b4380a5d89505417606, role.id=ac68e1820c604122981fd5c4e2ebc447, scope.project.id=2e832f037cf845379232686bfc78f98d) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:22.106215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a95db3-e6bc-43e2-b335-ca5ecb0a1681 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:22.108092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a95db3-e6bc-43e2-b335-ca5ecb0a1681 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:22.116168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 189/371] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Jul 31 00:54:21 2022] GET /identity/v3/role_assignments?group.id=41df40ce995b4b4380a5d89505417606&scope.project.id=2e832f037cf845379232686bfc78f98d&role.id=ac68e1820c604122981fd5c4e2ebc447 => generated 260 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:22.117785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a95db3-e6bc-43e2-b335-ca5ecb0a1681 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:22.248745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a95db3-e6bc-43e2-b335-ca5ecb0a1681 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:22.271867 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99391c9b-c497-4002-bff8-402c22af2237 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:22.292311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae2f3ba-69ec-4746-80de-dee1f8283041 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:22.293761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae2f3ba-69ec-4746-80de-dee1f8283041 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:22.294260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae2f3ba-69ec-4746-80de-dee1f8283041 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:22.294846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae2f3ba-69ec-4746-80de-dee1f8283041 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:22.350143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681b25aa-50bc-4fb1-a471-b624dfd89e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:22.350634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681b25aa-50bc-4fb1-a471-b624dfd89e84 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:22.350918 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681b25aa-50bc-4fb1-a471-b624dfd89e84 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:22.351193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681b25aa-50bc-4fb1-a471-b624dfd89e84 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:22.351982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681b25aa-50bc-4fb1-a471-b624dfd89e84 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:22.352311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681b25aa-50bc-4fb1-a471-b624dfd89e84 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:22.353747 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681b25aa-50bc-4fb1-a471-b624dfd89e84 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:22.365263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 183/372] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:22.372514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681b25aa-50bc-4fb1-a471-b624dfd89e84 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:22.377581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:22.403526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae2f3ba-69ec-4746-80de-dee1f8283041 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:22.405650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae2f3ba-69ec-4746-80de-dee1f8283041 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:22.450023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:22.529951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 190/373] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:22.537885 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae2f3ba-69ec-4746-80de-dee1f8283041 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:22.578654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681b25aa-50bc-4fb1-a471-b624dfd89e84 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:22.617052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221e4e0e-4dbc-419b-8b56-424e19801844 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:22.618006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221e4e0e-4dbc-419b-8b56-424e19801844 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:22.618138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221e4e0e-4dbc-419b-8b56-424e19801844 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:22.618610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221e4e0e-4dbc-419b-8b56-424e19801844 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:22.633667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:22.662219 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae2f3ba-69ec-4746-80de-dee1f8283041 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:22.709474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221e4e0e-4dbc-419b-8b56-424e19801844 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:22.710957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221e4e0e-4dbc-419b-8b56-424e19801844 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:22.715302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5be500-773f-4baa-8471-e9f1a7285540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:22.715986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5be500-773f-4baa-8471-e9f1a7285540 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:22.716262 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5be500-773f-4baa-8471-e9f1a7285540 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:22.716526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5be500-773f-4baa-8471-e9f1a7285540 None admin] PATH_INFO: `/v3/projects/18e627c596bb4c99b66fda211d0e4785` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:22.717736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5be500-773f-4baa-8471-e9f1a7285540 None admin] RBAC: Authorizing `identity:get_project(project_id=18e627c596bb4c99b66fda211d0e4785)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:22.719320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5be500-773f-4baa-8471-e9f1a7285540 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:22.726632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 191/374] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:22 2022] GET /identity/v3/projects/18e627c596bb4c99b66fda211d0e4785 => generated 344 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:22.728398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e47d16b-4ae4-4a95-9c9c-2339a71ce874 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:22.728693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e47d16b-4ae4-4a95-9c9c-2339a71ce874 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:22.729130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e47d16b-4ae4-4a95-9c9c-2339a71ce874 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:22.731062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 192/375] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:22.732426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e47d16b-4ae4-4a95-9c9c-2339a71ce874 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:22.762479 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:22.846105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 184/376] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 474 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:22.852193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8f7be4-a678-425e-af1f-9b763bca75db None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:22.852569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8f7be4-a678-425e-af1f-9b763bca75db None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:22.852829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8f7be4-a678-425e-af1f-9b763bca75db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:22.865257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e47d16b-4ae4-4a95-9c9c-2339a71ce874 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:22.873214 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ee8f7be4-a678-425e-af1f-9b763bca75db None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:22.874521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ee8f7be4-a678-425e-af1f-9b763bca75db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:22.909039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24302e4-991a-47b5-928f-78b0e9cd935f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:22.910050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24302e4-991a-47b5-928f-78b0e9cd935f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:22.910598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24302e4-991a-47b5-928f-78b0e9cd935f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:22.910881 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24302e4-991a-47b5-928f-78b0e9cd935f None admin] PATH_INFO: `/v3/groups/8e44f262cddd4b8a9abb0018814b900c` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:22.917285 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24302e4-991a-47b5-928f-78b0e9cd935f None admin] RBAC: Authorizing `identity:get_group(group_id=8e44f262cddd4b8a9abb0018814b900c)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:22.918907 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24302e4-991a-47b5-928f-78b0e9cd935f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:22.927556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 193/377] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:22 2022] GET /identity/v3/groups/8e44f262cddd4b8a9abb0018814b900c => generated 229 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:22.928502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24302e4-991a-47b5-928f-78b0e9cd935f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:22.944272 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:22.969958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 185/378] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:22.970872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8f7be4-a678-425e-af1f-9b763bca75db None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:23.080547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24302e4-991a-47b5-928f-78b0e9cd935f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:23.084907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:23.130407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba42c9f8-d503-4e34-9850-23e09ba66550 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:23.131080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba42c9f8-d503-4e34-9850-23e09ba66550 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:23.132147 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba42c9f8-d503-4e34-9850-23e09ba66550 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:23.132511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba42c9f8-d503-4e34-9850-23e09ba66550 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:23.133864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba42c9f8-d503-4e34-9850-23e09ba66550 None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:23.134725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8f7be4-a678-425e-af1f-9b763bca75db None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:23.135328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba42c9f8-d503-4e34-9850-23e09ba66550 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:23.137611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 194/379] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:22 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:23.138430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba42c9f8-d503-4e34-9850-23e09ba66550 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:23.187450 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40da6cf4-9ac0-44d8-8ff7-e88682daff1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:23.189363 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40da6cf4-9ac0-44d8-8ff7-e88682daff1c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:23.189646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40da6cf4-9ac0-44d8-8ff7-e88682daff1c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:23.189921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40da6cf4-9ac0-44d8-8ff7-e88682daff1c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:23.190723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40da6cf4-9ac0-44d8-8ff7-e88682daff1c None admin] RBAC: Adding query filter params (group.id=8e44f262cddd4b8a9abb0018814b900c, role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=18e627c596bb4c99b66fda211d0e4785) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:23.191031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40da6cf4-9ac0-44d8-8ff7-e88682daff1c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:23.192523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40da6cf4-9ac0-44d8-8ff7-e88682daff1c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:23.199119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 186/380] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Jul 31 00:54:22 2022] GET /identity/v3/role_assignments?group.id=8e44f262cddd4b8a9abb0018814b900c&scope.project.id=18e627c596bb4c99b66fda211d0e4785&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 260 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:23.200004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40da6cf4-9ac0-44d8-8ff7-e88682daff1c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:23.296719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:23.339826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba42c9f8-d503-4e34-9850-23e09ba66550 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:23.414966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf0112d-1a2f-4b01-9e6a-227189783b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:23.415746 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf0112d-1a2f-4b01-9e6a-227189783b25 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:23.416027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf0112d-1a2f-4b01-9e6a-227189783b25 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:23.416460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf0112d-1a2f-4b01-9e6a-227189783b25 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:23.433350 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf0112d-1a2f-4b01-9e6a-227189783b25 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-caf0112d-1a2f-4b01-9e6a-227189783b25 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:23.438648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:23.439613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:23.440405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 195/381] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:23.446897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf0112d-1a2f-4b01-9e6a-227189783b25 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:23.463246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40da6cf4-9ac0-44d8-8ff7-e88682daff1c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:23.534144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:23.550419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a23fd02-d1ef-4b6b-a8cb-0bc604bd92bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:23.551806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a23fd02-d1ef-4b6b-a8cb-0bc604bd92bd None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:23.552405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a23fd02-d1ef-4b6b-a8cb-0bc604bd92bd None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:23.553019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a23fd02-d1ef-4b6b-a8cb-0bc604bd92bd None admin] PATH_INFO: `/v3/groups/41df40ce995b4b4380a5d89505417606` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:23.554925 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a23fd02-d1ef-4b6b-a8cb-0bc604bd92bd None admin] RBAC: Authorizing `identity:get_group(group_id=41df40ce995b4b4380a5d89505417606)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:23.563390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a23fd02-d1ef-4b6b-a8cb-0bc604bd92bd None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:23.566641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 187/382] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:23 2022] GET /identity/v3/groups/41df40ce995b4b4380a5d89505417606 => generated 226 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:23.583586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a23fd02-d1ef-4b6b-a8cb-0bc604bd92bd None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:23.623014 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf0112d-1a2f-4b01-9e6a-227189783b25 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:23.661436 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e501e849-e3be-4711-83af-f3399636e696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:23.662211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e501e849-e3be-4711-83af-f3399636e696 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:23.662379 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e501e849-e3be-4711-83af-f3399636e696 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:23.662658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e501e849-e3be-4711-83af-f3399636e696 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:23.663405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e501e849-e3be-4711-83af-f3399636e696 None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:23.663714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e501e849-e3be-4711-83af-f3399636e696 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:23.665248 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e501e849-e3be-4711-83af-f3399636e696 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:23.671021 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:23.677727 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 196/383] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:23.691613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e501e849-e3be-4711-83af-f3399636e696 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:23.808358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a23fd02-d1ef-4b6b-a8cb-0bc604bd92bd None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:23.830226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e501e849-e3be-4711-83af-f3399636e696 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:23.852392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecefc5e-966c-491b-9e43-d0a42ea80ccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:23.853121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecefc5e-966c-491b-9e43-d0a42ea80ccf None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:23.853406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecefc5e-966c-491b-9e43-d0a42ea80ccf None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:23.853676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecefc5e-966c-491b-9e43-d0a42ea80ccf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:23.854832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecefc5e-966c-491b-9e43-d0a42ea80ccf None admin] RBAC: Adding query filter params (group.id=41df40ce995b4b4380a5d89505417606, role.id=0a2f5ca94533412088499eaccd7b2e21, scope.project.id=2e832f037cf845379232686bfc78f98d) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:23.855138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecefc5e-966c-491b-9e43-d0a42ea80ccf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:23.856812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecefc5e-966c-491b-9e43-d0a42ea80ccf None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:23.864004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 188/384] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Jul 31 00:54:23 2022] GET /identity/v3/role_assignments?group.id=41df40ce995b4b4380a5d89505417606&scope.project.id=2e832f037cf845379232686bfc78f98d&role.id=0a2f5ca94533412088499eaccd7b2e21 => generated 260 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:23.879515 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a451207-d7bc-4c29-b841-63eb3dfcb4cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:23.880487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a451207-d7bc-4c29-b841-63eb3dfcb4cb None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:23.880961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a451207-d7bc-4c29-b841-63eb3dfcb4cb None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:23.881438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a451207-d7bc-4c29-b841-63eb3dfcb4cb None admin] PATH_INFO: `/v3/groups/41df40ce995b4b4380a5d89505417606` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:23.888366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a451207-d7bc-4c29-b841-63eb3dfcb4cb None admin] RBAC: Authorizing `identity:get_group(group_id=41df40ce995b4b4380a5d89505417606)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:23.889909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a451207-d7bc-4c29-b841-63eb3dfcb4cb None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:23.892228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 197/385] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:23 2022] GET /identity/v3/groups/41df40ce995b4b4380a5d89505417606 => generated 226 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:23.897989 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecefc5e-966c-491b-9e43-d0a42ea80ccf None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:23.986707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:23.997413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65151142-23c7-44b2-99d8-4d64a1e39156 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:23.997696 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65151142-23c7-44b2-99d8-4d64a1e39156 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:23.997972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65151142-23c7-44b2-99d8-4d64a1e39156 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:23.999509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 198/386] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:54:24.005782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c6bfed-f91c-4b67-b9ab-681863aa47f4 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:24.006131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c6bfed-f91c-4b67-b9ab-681863aa47f4 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:24.006330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c6bfed-f91c-4b67-b9ab-681863aa47f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:24.041096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-67c6bfed-f91c-4b67-b9ab-681863aa47f4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:24.042361 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-67c6bfed-f91c-4b67-b9ab-681863aa47f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:24.105590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecefc5e-966c-491b-9e43-d0a42ea80ccf None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:24.143801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:24.151307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 199/387] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:54:24.173231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8028639d-d00c-4a94-9578-89eb9e5605e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:24.174441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8028639d-d00c-4a94-9578-89eb9e5605e0 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:24.175518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c6bfed-f91c-4b67-b9ab-681863aa47f4 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:24.184117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8028639d-d00c-4a94-9578-89eb9e5605e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:24.184606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8028639d-d00c-4a94-9578-89eb9e5605e0 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:24.186439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8028639d-d00c-4a94-9578-89eb9e5605e0 None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:24.187960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8028639d-d00c-4a94-9578-89eb9e5605e0 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:24.190034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 189/388] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:23 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:24.201052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8028639d-d00c-4a94-9578-89eb9e5605e0 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:24.339675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c6bfed-f91c-4b67-b9ab-681863aa47f4 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:24.389176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfeb59b6-ef87-4837-aa4f-bb552fc83c02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:24.389844 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfeb59b6-ef87-4837-aa4f-bb552fc83c02 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:24.390290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfeb59b6-ef87-4837-aa4f-bb552fc83c02 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:24.390570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfeb59b6-ef87-4837-aa4f-bb552fc83c02 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:24.413722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfeb59b6-ef87-4837-aa4f-bb552fc83c02 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-dfeb59b6-ef87-4837-aa4f-bb552fc83c02 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:24.419290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:24.420241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:24.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 200/389] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:54:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:24.429393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfeb59b6-ef87-4837-aa4f-bb552fc83c02 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:24.443805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8028639d-d00c-4a94-9578-89eb9e5605e0 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:24.502249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:24.515420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c15fed-8547-45ae-af7d-98162ce16817 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:24.516120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c15fed-8547-45ae-af7d-98162ce16817 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:24.516402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c15fed-8547-45ae-af7d-98162ce16817 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:24.516836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c15fed-8547-45ae-af7d-98162ce16817 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34/groups/41df40ce995b4b4380a5d89505417606/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:24.525763 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c15fed-8547-45ae-af7d-98162ce16817 None admin] RBAC: Authorizing `identity:create_grant(project_id=c0935b3a5a454c70bf75163763e11b34, group_id=41df40ce995b4b4380a5d89505417606, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:24.532127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c15fed-8547-45ae-af7d-98162ce16817 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:24.557419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:24.563751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 190/390] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Jul 31 00:54:24 2022] PUT /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34/groups/41df40ce995b4b4380a5d89505417606/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:24.624931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfeb59b6-ef87-4837-aa4f-bb552fc83c02 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:24.671110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688cb54e-25a7-4330-91bf-68d0a4f2dc5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:24.672147 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688cb54e-25a7-4330-91bf-68d0a4f2dc5a None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:24.672147 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688cb54e-25a7-4330-91bf-68d0a4f2dc5a None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:24.672381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688cb54e-25a7-4330-91bf-68d0a4f2dc5a None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:24.673012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688cb54e-25a7-4330-91bf-68d0a4f2dc5a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:24.673314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688cb54e-25a7-4330-91bf-68d0a4f2dc5a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:24.675309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688cb54e-25a7-4330-91bf-68d0a4f2dc5a None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:24.686182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 201/391] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:24.694921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c15fed-8547-45ae-af7d-98162ce16817 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:24.890460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c15fed-8547-45ae-af7d-98162ce16817 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:24.955988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16fe036-10f8-435b-826d-3564e8e438d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:24.957512 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16fe036-10f8-435b-826d-3564e8e438d3 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:24.958247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16fe036-10f8-435b-826d-3564e8e438d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:24.958955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16fe036-10f8-435b-826d-3564e8e438d3 None admin] PATH_INFO: `/v3/groups/41df40ce995b4b4380a5d89505417606` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:24.961444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16fe036-10f8-435b-826d-3564e8e438d3 None admin] RBAC: Authorizing `identity:get_group(group_id=41df40ce995b4b4380a5d89505417606)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:24.964502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fed81c7-7b56-4d57-b0fc-eac2b2f02571 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:24.964803 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fed81c7-7b56-4d57-b0fc-eac2b2f02571 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:24.965064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fed81c7-7b56-4d57-b0fc-eac2b2f02571 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:24.965780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16fe036-10f8-435b-826d-3564e8e438d3 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:24.967786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 202/392] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:54:24.969782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 191/393] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:24 2022] GET /identity/v3/groups/41df40ce995b4b4380a5d89505417606 => generated 226 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:24.979508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7adb9b-26bc-4bc7-a1b1-1f059a6d9058 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:24.979739 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16fe036-10f8-435b-826d-3564e8e438d3 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:24.979862 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7adb9b-26bc-4bc7-a1b1-1f059a6d9058 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:24.980207 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7adb9b-26bc-4bc7-a1b1-1f059a6d9058 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:25.019436 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ad7adb9b-26bc-4bc7-a1b1-1f059a6d9058 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:25.020683 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ad7adb9b-26bc-4bc7-a1b1-1f059a6d9058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:25.091763 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:25.124583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 203/394] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:25.148794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7adb9b-26bc-4bc7-a1b1-1f059a6d9058 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:25.173277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16fe036-10f8-435b-826d-3564e8e438d3 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:25.209387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:25.237558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cbe68b-a77a-448e-9b19-bc2788f9a649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:25.238353 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cbe68b-a77a-448e-9b19-bc2788f9a649 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:25.238736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cbe68b-a77a-448e-9b19-bc2788f9a649 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:25.239110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cbe68b-a77a-448e-9b19-bc2788f9a649 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:25.241197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cbe68b-a77a-448e-9b19-bc2788f9a649 None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:25.242785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cbe68b-a77a-448e-9b19-bc2788f9a649 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:25.245489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 192/395] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:24 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:25.254883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cbe68b-a77a-448e-9b19-bc2788f9a649 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:25.283813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7adb9b-26bc-4bc7-a1b1-1f059a6d9058 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:25.332451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ffee6f-c801-495f-ab8a-03faa6bdf49c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:25.333114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ffee6f-c801-495f-ab8a-03faa6bdf49c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:25.333388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ffee6f-c801-495f-ab8a-03faa6bdf49c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:25.333693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ffee6f-c801-495f-ab8a-03faa6bdf49c None admin] PATH_INFO: `/v3/roles/member` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:25.357882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ffee6f-c801-495f-ab8a-03faa6bdf49c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-23ffee6f-c801-495f-ab8a-03faa6bdf49c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:25.368990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:25.369923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:25.370728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 204/396] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:25.381869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ffee6f-c801-495f-ab8a-03faa6bdf49c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:25.463160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cbe68b-a77a-448e-9b19-bc2788f9a649 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:25.499745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:25.513180 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a853f1b2-8f5e-46bc-8607-7ba0c12e0984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:25.513890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a853f1b2-8f5e-46bc-8607-7ba0c12e0984 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:25.514172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a853f1b2-8f5e-46bc-8607-7ba0c12e0984 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:25.519379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a853f1b2-8f5e-46bc-8607-7ba0c12e0984 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34/groups/41df40ce995b4b4380a5d89505417606/roles/0a2f5ca94533412088499eaccd7b2e21` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:25.519379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a853f1b2-8f5e-46bc-8607-7ba0c12e0984 None admin] RBAC: Authorizing `identity:create_grant(project_id=c0935b3a5a454c70bf75163763e11b34, group_id=41df40ce995b4b4380a5d89505417606, role_id=0a2f5ca94533412088499eaccd7b2e21)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:25.519379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a853f1b2-8f5e-46bc-8607-7ba0c12e0984 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:25.567532 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 193/397] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Jul 31 00:54:25 2022] PUT /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34/groups/41df40ce995b4b4380a5d89505417606/roles/0a2f5ca94533412088499eaccd7b2e21 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 00:54:25.591048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ffee6f-c801-495f-ab8a-03faa6bdf49c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:25.606162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:25.636151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b614f7-3076-4511-9603-1ee636065026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:25.636775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b614f7-3076-4511-9603-1ee636065026 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:25.637053 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b614f7-3076-4511-9603-1ee636065026 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:25.637377 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b614f7-3076-4511-9603-1ee636065026 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:25.638111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b614f7-3076-4511-9603-1ee636065026 None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:25.638411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b614f7-3076-4511-9603-1ee636065026 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:25.639880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b614f7-3076-4511-9603-1ee636065026 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:25.653501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 205/398] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:25.656462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a853f1b2-8f5e-46bc-8607-7ba0c12e0984 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:25.807467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a853f1b2-8f5e-46bc-8607-7ba0c12e0984 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:25.833885 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5849409c-d4a0-4e6a-aed6-270d8e1d80b0 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:25.834432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5849409c-d4a0-4e6a-aed6-270d8e1d80b0 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:25.834901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5849409c-d4a0-4e6a-aed6-270d8e1d80b0 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:25.837486 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 206/399] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:25.843277 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0435bfbb-6348-4b32-a5f4-1b351fc67d71 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:25.843679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0435bfbb-6348-4b32-a5f4-1b351fc67d71 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:25.843852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0435bfbb-6348-4b32-a5f4-1b351fc67d71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:25.849394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a04afe9-1494-4aed-b397-bc5deec9124e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:25.850080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a04afe9-1494-4aed-b397-bc5deec9124e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:25.850360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a04afe9-1494-4aed-b397-bc5deec9124e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:25.850634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a04afe9-1494-4aed-b397-bc5deec9124e None admin] PATH_INFO: `/v3/groups/41df40ce995b4b4380a5d89505417606` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:25.851898 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a04afe9-1494-4aed-b397-bc5deec9124e None admin] RBAC: Authorizing `identity:get_group(group_id=41df40ce995b4b4380a5d89505417606)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:25.853408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a04afe9-1494-4aed-b397-bc5deec9124e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:25.856050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 194/400] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:25 2022] GET /identity/v3/groups/41df40ce995b4b4380a5d89505417606 => generated 226 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:25.862037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a04afe9-1494-4aed-b397-bc5deec9124e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:25.905053 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-0435bfbb-6348-4b32-a5f4-1b351fc67d71 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:25.907073 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0435bfbb-6348-4b32-a5f4-1b351fc67d71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:25.931774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:25.999019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a04afe9-1494-4aed-b397-bc5deec9124e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:26.015972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 207/401] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:54:26.043217 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598f1dac-68e5-4b7d-9511-7c4b83f68d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:26.044135 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598f1dac-68e5-4b7d-9511-7c4b83f68d94 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:26.044556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598f1dac-68e5-4b7d-9511-7c4b83f68d94 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:26.044992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598f1dac-68e5-4b7d-9511-7c4b83f68d94 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:26.046052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0435bfbb-6348-4b32-a5f4-1b351fc67d71 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:26.046663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598f1dac-68e5-4b7d-9511-7c4b83f68d94 None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:26.048411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598f1dac-68e5-4b7d-9511-7c4b83f68d94 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:26.050700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 195/402] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:25 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:26.068828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598f1dac-68e5-4b7d-9511-7c4b83f68d94 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:26.163298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0435bfbb-6348-4b32-a5f4-1b351fc67d71 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:26.165070 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:26.227786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598f1dac-68e5-4b7d-9511-7c4b83f68d94 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:26.257943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:26.272730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce854938-dcfe-4136-8df6-b1520d8dc38e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:26.273425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce854938-dcfe-4136-8df6-b1520d8dc38e None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:26.273725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce854938-dcfe-4136-8df6-b1520d8dc38e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:26.273973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce854938-dcfe-4136-8df6-b1520d8dc38e None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d/groups/41df40ce995b4b4380a5d89505417606/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:26.277980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce854938-dcfe-4136-8df6-b1520d8dc38e None admin] RBAC: Authorizing `identity:create_grant(project_id=2e832f037cf845379232686bfc78f98d, group_id=41df40ce995b4b4380a5d89505417606, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:26.279453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce854938-dcfe-4136-8df6-b1520d8dc38e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:26.286847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4389d9-7729-4678-afaa-17c7d57cd27a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:26.287973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4389d9-7729-4678-afaa-17c7d57cd27a None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:26.288465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4389d9-7729-4678-afaa-17c7d57cd27a None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:26.288955 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4389d9-7729-4678-afaa-17c7d57cd27a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:26.297158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 196/403] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Jul 31 00:54:26 2022] PUT /identity/v3/projects/2e832f037cf845379232686bfc78f98d/groups/41df40ce995b4b4380a5d89505417606/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:26.321144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4389d9-7729-4678-afaa-17c7d57cd27a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-5e4389d9-7729-4678-afaa-17c7d57cd27a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:26.326581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:26.327498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:26.328577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 208/404] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:54:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:26.334626 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce854938-dcfe-4136-8df6-b1520d8dc38e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:26.461325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce854938-dcfe-4136-8df6-b1520d8dc38e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:26.501288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481e0986-bada-4d16-ba0a-7b727b916f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:26.502542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481e0986-bada-4d16-ba0a-7b727b916f6f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:26.502828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481e0986-bada-4d16-ba0a-7b727b916f6f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:26.503099 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481e0986-bada-4d16-ba0a-7b727b916f6f None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:26.503889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481e0986-bada-4d16-ba0a-7b727b916f6f None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:26.504194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481e0986-bada-4d16-ba0a-7b727b916f6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:26.505575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481e0986-bada-4d16-ba0a-7b727b916f6f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:26.515469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 197/405] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:26.520893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4389d9-7729-4678-afaa-17c7d57cd27a None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:26.650843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4389d9-7729-4678-afaa-17c7d57cd27a None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:26.692012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abd3568-1ef4-4ec4-90d7-ac1c699d23ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:26.692664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abd3568-1ef4-4ec4-90d7-ac1c699d23ad None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:26.692941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abd3568-1ef4-4ec4-90d7-ac1c699d23ad None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:26.693213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abd3568-1ef4-4ec4-90d7-ac1c699d23ad None admin] PATH_INFO: `/v3/groups/8e44f262cddd4b8a9abb0018814b900c` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:26.694438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abd3568-1ef4-4ec4-90d7-ac1c699d23ad None admin] RBAC: Authorizing `identity:get_group(group_id=8e44f262cddd4b8a9abb0018814b900c)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:26.695994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abd3568-1ef4-4ec4-90d7-ac1c699d23ad None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:26.698001 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 209/406] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:26 2022] GET /identity/v3/groups/8e44f262cddd4b8a9abb0018814b900c => generated 229 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:26.703392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481e0986-bada-4d16-ba0a-7b727b916f6f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:26.758535 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:26.822283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481e0986-bada-4d16-ba0a-7b727b916f6f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:26.860184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5693f2a-1e12-48c4-801d-753566a6a11d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:26.860828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5693f2a-1e12-48c4-801d-753566a6a11d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:26.861713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5693f2a-1e12-48c4-801d-753566a6a11d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:26.862003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5693f2a-1e12-48c4-801d-753566a6a11d None admin] PATH_INFO: `/v3/projects/18e627c596bb4c99b66fda211d0e4785` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:26.863196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5693f2a-1e12-48c4-801d-753566a6a11d None admin] RBAC: Authorizing `identity:get_project(project_id=18e627c596bb4c99b66fda211d0e4785)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:26.864792 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5693f2a-1e12-48c4-801d-753566a6a11d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:26.866837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 198/407] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:26 2022] GET /identity/v3/projects/18e627c596bb4c99b66fda211d0e4785 => generated 344 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:26.872309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abd3568-1ef4-4ec4-90d7-ac1c699d23ad None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:26.885448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b867c15a-527e-4f50-bcaf-56e4c69566e9 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:26.885773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b867c15a-527e-4f50-bcaf-56e4c69566e9 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:26.886004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b867c15a-527e-4f50-bcaf-56e4c69566e9 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:26.891038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 199/408] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:26 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:26.892633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1061efcb-c24c-478f-b0b4-270a83237042 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:26.892912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1061efcb-c24c-478f-b0b4-270a83237042 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:26.893183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1061efcb-c24c-478f-b0b4-270a83237042 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:26.938002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:26.942296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-1061efcb-c24c-478f-b0b4-270a83237042 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:26.948330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1061efcb-c24c-478f-b0b4-270a83237042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:27.019598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abd3568-1ef4-4ec4-90d7-ac1c699d23ad None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:27.059946 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10d3e5a-bde3-4c34-a505-e486debe1f60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:27.060650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10d3e5a-bde3-4c34-a505-e486debe1f60 None admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:27.060932 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10d3e5a-bde3-4c34-a505-e486debe1f60 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:27.061254 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10d3e5a-bde3-4c34-a505-e486debe1f60 None admin] PATH_INFO: `/v3/projects/18e627c596bb4c99b66fda211d0e4785/groups/8e44f262cddd4b8a9abb0018814b900c/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:27.063545 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10d3e5a-bde3-4c34-a505-e486debe1f60 None admin] RBAC: Authorizing `identity:create_grant(project_id=18e627c596bb4c99b66fda211d0e4785, group_id=8e44f262cddd4b8a9abb0018814b900c, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:27.065385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10d3e5a-bde3-4c34-a505-e486debe1f60 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:27.082968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 210/409] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Jul 31 00:54:26 2022] PUT /identity/v3/projects/18e627c596bb4c99b66fda211d0e4785/groups/8e44f262cddd4b8a9abb0018814b900c/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:27.169639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 200/410] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 278 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:27.204193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10d3e5a-bde3-4c34-a505-e486debe1f60 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:27.328148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:27.339252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10d3e5a-bde3-4c34-a505-e486debe1f60 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:27.382635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd45feb2-4863-42b7-b902-1f847350fea0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:27.383291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd45feb2-4863-42b7-b902-1f847350fea0 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:27.383598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd45feb2-4863-42b7-b902-1f847350fea0 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:27.383858 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd45feb2-4863-42b7-b902-1f847350fea0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:27.397002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd45feb2-4863-42b7-b902-1f847350fea0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-dd45feb2-4863-42b7-b902-1f847350fea0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:27.402206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:27.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:27.404169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 211/411] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:54:27 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:27.410386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1061efcb-c24c-478f-b0b4-270a83237042 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:27.500249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:27.578046 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1061efcb-c24c-478f-b0b4-270a83237042 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:27.638647 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58ebb38-e878-44b8-95ef-d7a5adde694f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:27.639386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58ebb38-e878-44b8-95ef-d7a5adde694f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:27.639649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58ebb38-e878-44b8-95ef-d7a5adde694f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:27.639920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58ebb38-e878-44b8-95ef-d7a5adde694f None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:27.640641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58ebb38-e878-44b8-95ef-d7a5adde694f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:27.640940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58ebb38-e878-44b8-95ef-d7a5adde694f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:27.642487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58ebb38-e878-44b8-95ef-d7a5adde694f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:27.652379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 201/412] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:27.659998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd45feb2-4863-42b7-b902-1f847350fea0 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:27.925821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd45feb2-4863-42b7-b902-1f847350fea0 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:27.980241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d09cefa-8163-4ccd-8242-32821e65c1a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:27.981175 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d09cefa-8163-4ccd-8242-32821e65c1a7 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:27.981634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d09cefa-8163-4ccd-8242-32821e65c1a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:27.982082 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d09cefa-8163-4ccd-8242-32821e65c1a7 None admin] PATH_INFO: `/v3/groups/41df40ce995b4b4380a5d89505417606` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:27.983641 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d09cefa-8163-4ccd-8242-32821e65c1a7 None admin] RBAC: Authorizing `identity:get_group(group_id=41df40ce995b4b4380a5d89505417606)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:27.985374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d09cefa-8163-4ccd-8242-32821e65c1a7 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:27.987845 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 212/413] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:27 2022] GET /identity/v3/groups/41df40ce995b4b4380a5d89505417606 => generated 226 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:27.993423 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58ebb38-e878-44b8-95ef-d7a5adde694f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:28.065010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd2572-5540-4dda-84bf-d9c2e7b5d9e0 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:28.065567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd2572-5540-4dda-84bf-d9c2e7b5d9e0 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:28.066043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd2572-5540-4dda-84bf-d9c2e7b5d9e0 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:28.071530 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 213/414] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:28 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:54:28.078399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8aaa5e-4440-4535-b1b2-3fdb86b690a1 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:28.078998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8aaa5e-4440-4535-b1b2-3fdb86b690a1 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:28.079524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8aaa5e-4440-4535-b1b2-3fdb86b690a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:28.118552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-8b8aaa5e-4440-4535-b1b2-3fdb86b690a1 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:28.120644 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8b8aaa5e-4440-4535-b1b2-3fdb86b690a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:28.171795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:28.172705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58ebb38-e878-44b8-95ef-d7a5adde694f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:28.175797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:28.215019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee30318-2c39-4b0c-a494-c8d169fea43b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:28.216028 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee30318-2c39-4b0c-a494-c8d169fea43b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:28.216507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee30318-2c39-4b0c-a494-c8d169fea43b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:28.216991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee30318-2c39-4b0c-a494-c8d169fea43b None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:28.218396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee30318-2c39-4b0c-a494-c8d169fea43b None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:28.220106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee30318-2c39-4b0c-a494-c8d169fea43b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:28.223281 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 202/415] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:27 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:28.229282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee30318-2c39-4b0c-a494-c8d169fea43b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:28.346254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee30318-2c39-4b0c-a494-c8d169fea43b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:28.391906 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db62b48-f2e2-48fe-aa94-394ea855eea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:28.394283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db62b48-f2e2-48fe-aa94-394ea855eea5 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:28.394907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db62b48-f2e2-48fe-aa94-394ea855eea5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:28.395430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db62b48-f2e2-48fe-aa94-394ea855eea5 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d/groups/41df40ce995b4b4380a5d89505417606/roles/0a2f5ca94533412088499eaccd7b2e21` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:28.398300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db62b48-f2e2-48fe-aa94-394ea855eea5 None admin] RBAC: Authorizing `identity:create_grant(project_id=2e832f037cf845379232686bfc78f98d, group_id=41df40ce995b4b4380a5d89505417606, role_id=0a2f5ca94533412088499eaccd7b2e21)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:28.400322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db62b48-f2e2-48fe-aa94-394ea855eea5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:28.424697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 203/416] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Jul 31 00:54:28 2022] PUT /identity/v3/projects/2e832f037cf845379232686bfc78f98d/groups/41df40ce995b4b4380a5d89505417606/roles/0a2f5ca94533412088499eaccd7b2e21 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:28.441559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 214/417] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 364 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:28.478869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db62b48-f2e2-48fe-aa94-394ea855eea5 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:28.525501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:28.612624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db62b48-f2e2-48fe-aa94-394ea855eea5 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:28.651278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f2bcab-1bef-4096-922e-57f081db7a76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:28.651982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f2bcab-1bef-4096-922e-57f081db7a76 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:28.652458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f2bcab-1bef-4096-922e-57f081db7a76 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:28.652879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f2bcab-1bef-4096-922e-57f081db7a76 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:28.666281 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f2bcab-1bef-4096-922e-57f081db7a76 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-e7f2bcab-1bef-4096-922e-57f081db7a76 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:28.671744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:28.672732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:28.676735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 204/418] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:28.684296 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8aaa5e-4440-4535-b1b2-3fdb86b690a1 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:28.808594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8aaa5e-4440-4535-b1b2-3fdb86b690a1 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:28.846787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa81f1a-f59f-44aa-97c1-d994c7f503fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:28.847796 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa81f1a-f59f-44aa-97c1-d994c7f503fb None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:28.848372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa81f1a-f59f-44aa-97c1-d994c7f503fb None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:28.848976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa81f1a-f59f-44aa-97c1-d994c7f503fb None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:28.850508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa81f1a-f59f-44aa-97c1-d994c7f503fb None admin] RBAC: Adding query filter params (name=member) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:28.851123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa81f1a-f59f-44aa-97c1-d994c7f503fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:28.853553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa81f1a-f59f-44aa-97c1-d994c7f503fb None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:28.866730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 215/419] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:28.872285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f2bcab-1bef-4096-922e-57f081db7a76 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:29.013519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:29.024433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f2bcab-1bef-4096-922e-57f081db7a76 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:29.068857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17d22df-a1f6-43aa-923f-82548e1a154e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:29.069915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17d22df-a1f6-43aa-923f-82548e1a154e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:29.070432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17d22df-a1f6-43aa-923f-82548e1a154e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:29.070908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17d22df-a1f6-43aa-923f-82548e1a154e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:29.161328 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17d22df-a1f6-43aa-923f-82548e1a154e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:29.163302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17d22df-a1f6-43aa-923f-82548e1a154e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:29.228900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47056cac-6bcc-4baf-9b9f-d1a394aa1746 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:29.229908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47056cac-6bcc-4baf-9b9f-d1a394aa1746 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:29.230356 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47056cac-6bcc-4baf-9b9f-d1a394aa1746 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:29.232860 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 216/420] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:29 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:54:29.239381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8092a1-1b30-41f5-bdd5-a0e22da676f2 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:29.239381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8092a1-1b30-41f5-bdd5-a0e22da676f2 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:29.239381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8092a1-1b30-41f5-bdd5-a0e22da676f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:29.288761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 205/421] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:29.289410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:29.292560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-7c8092a1-1b30-41f5-bdd5-a0e22da676f2 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:29.293949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7c8092a1-1b30-41f5-bdd5-a0e22da676f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:29.294571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17d22df-a1f6-43aa-923f-82548e1a154e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:29.311768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:29.414713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17d22df-a1f6-43aa-923f-82548e1a154e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:29.442130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 217/422] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:54:29.444023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78a700e-43af-4593-993f-d4c5cb7bf853 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:29.444676 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78a700e-43af-4593-993f-d4c5cb7bf853 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:29.445153 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78a700e-43af-4593-993f-d4c5cb7bf853 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:29.448190 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 218/423] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:29 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:29.459381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946a4787-5fed-4330-87c2-d169e7221242 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:29.459381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946a4787-5fed-4330-87c2-d169e7221242 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:29.459381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946a4787-5fed-4330-87c2-d169e7221242 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:29.461197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580a9769-e2db-4ffa-807d-de9bac161a25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:29.461862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580a9769-e2db-4ffa-807d-de9bac161a25 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:29.462131 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580a9769-e2db-4ffa-807d-de9bac161a25 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:29.462396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580a9769-e2db-4ffa-807d-de9bac161a25 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:29.463637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580a9769-e2db-4ffa-807d-de9bac161a25 None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:29.465093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580a9769-e2db-4ffa-807d-de9bac161a25 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:29.466818 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 206/424] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:29 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:29.470793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580a9769-e2db-4ffa-807d-de9bac161a25 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:29.492987 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-946a4787-5fed-4330-87c2-d169e7221242 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:29.494632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-946a4787-5fed-4330-87c2-d169e7221242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:29.575637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 219/425] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Jul 31 00:54:29.576994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946a4787-5fed-4330-87c2-d169e7221242 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:29.603563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:29.671405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580a9769-e2db-4ffa-807d-de9bac161a25 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:29.675880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:29.702055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946a4787-5fed-4330-87c2-d169e7221242 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:29.726222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a456a8a-ec17-459b-9db3-9f479270610d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:29.727287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a456a8a-ec17-459b-9db3-9f479270610d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:29.727821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a456a8a-ec17-459b-9db3-9f479270610d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:29.728316 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a456a8a-ec17-459b-9db3-9f479270610d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:29.745162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a456a8a-ec17-459b-9db3-9f479270610d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:29.748608 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fb1073-b717-4fee-a351-08adb33fc062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:29.749567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fb1073-b717-4fee-a351-08adb33fc062 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:29.750090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fb1073-b717-4fee-a351-08adb33fc062 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:29.750560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fb1073-b717-4fee-a351-08adb33fc062 None admin] PATH_INFO: `/v3/groups/41df40ce995b4b4380a5d89505417606` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:29.752830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fb1073-b717-4fee-a351-08adb33fc062 None admin] RBAC: Authorizing `identity:get_group(group_id=41df40ce995b4b4380a5d89505417606)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:29.754666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fb1073-b717-4fee-a351-08adb33fc062 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:29.756892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 220/426] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:29 2022] GET /identity/v3/groups/41df40ce995b4b4380a5d89505417606 => generated 226 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:29.757871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fb1073-b717-4fee-a351-08adb33fc062 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-9a456a8a-ec17-459b-9db3-9f479270610d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:29.758631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:29.759950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:29.761478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:29.761478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:29.761478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:29.761478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:29.761478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:29.761478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:29.761478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:29.761478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:29.761478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:29.761478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:29.761478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:29.761478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:29.771532 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 207/427] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:54:29 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:29.772903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a456a8a-ec17-459b-9db3-9f479270610d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:29.878111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fb1073-b717-4fee-a351-08adb33fc062 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:29.896517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a456a8a-ec17-459b-9db3-9f479270610d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:29.917626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76b9435-98f7-4bab-8310-67415dbb2986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:29.918494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76b9435-98f7-4bab-8310-67415dbb2986 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:29.918884 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76b9435-98f7-4bab-8310-67415dbb2986 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:29.919297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76b9435-98f7-4bab-8310-67415dbb2986 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:29.921278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:29.930066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:29.935769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76b9435-98f7-4bab-8310-67415dbb2986 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:29.941804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f2fbeb-5f12-4200-bb29-d7043a34bd0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-b76b9435-98f7-4bab-8310-67415dbb2986 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:29.942286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:29.943311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:29.944228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 221/428] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:29.944869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76b9435-98f7-4bab-8310-67415dbb2986 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:29.945739 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f2fbeb-5f12-4200-bb29-d7043a34bd0b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:29.946232 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f2fbeb-5f12-4200-bb29-d7043a34bd0b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:29.946697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f2fbeb-5f12-4200-bb29-d7043a34bd0b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:29.947724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f2fbeb-5f12-4200-bb29-d7043a34bd0b None admin] RBAC: Adding query filter params (group.id=41df40ce995b4b4380a5d89505417606, role.id=ac68e1820c604122981fd5c4e2ebc447, scope.project.id=c0935b3a5a454c70bf75163763e11b34) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:29.948283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f2fbeb-5f12-4200-bb29-d7043a34bd0b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:29.950122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f2fbeb-5f12-4200-bb29-d7043a34bd0b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:29.957014 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 208/429] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Jul 31 00:54:29 2022] GET /identity/v3/role_assignments?group.id=41df40ce995b4b4380a5d89505417606&scope.project.id=c0935b3a5a454c70bf75163763e11b34&role.id=ac68e1820c604122981fd5c4e2ebc447 => generated 611 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:29.958145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f2fbeb-5f12-4200-bb29-d7043a34bd0b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:30.106782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76b9435-98f7-4bab-8310-67415dbb2986 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:30.107230 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f2fbeb-5f12-4200-bb29-d7043a34bd0b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:30.153614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a51a9b0-c665-477d-a691-637d6780b727 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:30.154419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a51a9b0-c665-477d-a691-637d6780b727 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:30.154516 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c33d5a8-0025-4656-a5c5-108375618e7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:30.154741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a51a9b0-c665-477d-a691-637d6780b727 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:30.155036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a51a9b0-c665-477d-a691-637d6780b727 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:30.155137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c33d5a8-0025-4656-a5c5-108375618e7f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:30.155451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c33d5a8-0025-4656-a5c5-108375618e7f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:30.155677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c33d5a8-0025-4656-a5c5-108375618e7f None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:30.155771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a51a9b0-c665-477d-a691-637d6780b727 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:30.156074 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a51a9b0-c665-477d-a691-637d6780b727 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:30.156441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c33d5a8-0025-4656-a5c5-108375618e7f None admin] RBAC: Adding query filter params (name=member) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:30.156749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c33d5a8-0025-4656-a5c5-108375618e7f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:30.157517 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a51a9b0-c665-477d-a691-637d6780b727 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:30.158744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c33d5a8-0025-4656-a5c5-108375618e7f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:30.169116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 222/430] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:29 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:30.169302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 209/431] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:54:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:30.170491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847a0778-e93b-4b16-ac61-1de2c71e1298 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:30.170691 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847a0778-e93b-4b16-ac61-1de2c71e1298 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:30.170966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847a0778-e93b-4b16-ac61-1de2c71e1298 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:30.172489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 223/432] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:30.176920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c33d5a8-0025-4656-a5c5-108375618e7f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:30.182739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61fa9d5-7633-4b7b-b967-0ed873b45755 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:30.183086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61fa9d5-7633-4b7b-b967-0ed873b45755 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:30.184138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61fa9d5-7633-4b7b-b967-0ed873b45755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:30.205971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-d61fa9d5-7633-4b7b-b967-0ed873b45755 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:30.207546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d61fa9d5-7633-4b7b-b967-0ed873b45755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:30.230875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:30.292041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:30.306989 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 224/433] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:30.309986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c33d5a8-0025-4656-a5c5-108375618e7f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:30.310270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61fa9d5-7633-4b7b-b967-0ed873b45755 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:30.373435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2b34db-12a6-4f5c-adf4-17be0bf50d74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:30.374352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2b34db-12a6-4f5c-adf4-17be0bf50d74 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:30.375346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2b34db-12a6-4f5c-adf4-17be0bf50d74 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:30.375822 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2b34db-12a6-4f5c-adf4-17be0bf50d74 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:30.446841 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61fa9d5-7633-4b7b-b967-0ed873b45755 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:30.492252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2b34db-12a6-4f5c-adf4-17be0bf50d74 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:30.493872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2b34db-12a6-4f5c-adf4-17be0bf50d74 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:30.511061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cc7642-0a8a-4c25-a802-dae142570382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:30.512025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cc7642-0a8a-4c25-a802-dae142570382 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:30.512442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cc7642-0a8a-4c25-a802-dae142570382 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:30.512875 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cc7642-0a8a-4c25-a802-dae142570382 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:30.569077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:30.602985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:30.621704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cc7642-0a8a-4c25-a802-dae142570382 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:30.624197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cc7642-0a8a-4c25-a802-dae142570382 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:30.663402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 210/434] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 482 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:30.664358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2b34db-12a6-4f5c-adf4-17be0bf50d74 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:30.765640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 225/435] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:30.766843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cc7642-0a8a-4c25-a802-dae142570382 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:30.810381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2b34db-12a6-4f5c-adf4-17be0bf50d74 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:30.853065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20c0d78-644c-4adc-86f6-610a5c222d61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:30.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20c0d78-644c-4adc-86f6-610a5c222d61 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:30.854255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20c0d78-644c-4adc-86f6-610a5c222d61 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:30.854552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20c0d78-644c-4adc-86f6-610a5c222d61 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:30.867446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20c0d78-644c-4adc-86f6-610a5c222d61 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:30.867895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-e20c0d78-644c-4adc-86f6-610a5c222d61 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:30.873857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:30.875150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:30.876318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 211/436] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:54:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:30.881273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20c0d78-644c-4adc-86f6-610a5c222d61 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:30.887830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cc7642-0a8a-4c25-a802-dae142570382 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:30.893568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:30.943935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41386b09-ef9c-4bba-bd4d-651ffae141a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:30.945004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41386b09-ef9c-4bba-bd4d-651ffae141a6 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:30.945492 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41386b09-ef9c-4bba-bd4d-651ffae141a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:30.945962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41386b09-ef9c-4bba-bd4d-651ffae141a6 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:30.947768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41386b09-ef9c-4bba-bd4d-651ffae141a6 None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:30.949833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41386b09-ef9c-4bba-bd4d-651ffae141a6 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:30.958908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 226/437] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:30 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:30.958908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41386b09-ef9c-4bba-bd4d-651ffae141a6 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:31.012660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20c0d78-644c-4adc-86f6-610a5c222d61 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:31.057241 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466277c9-2ec9-4b80-94b7-691b30d2aac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:31.057754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466277c9-2ec9-4b80-94b7-691b30d2aac5 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:31.058049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466277c9-2ec9-4b80-94b7-691b30d2aac5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:31.058316 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466277c9-2ec9-4b80-94b7-691b30d2aac5 None admin] PATH_INFO: `/v3/projects/c0935b3a5a454c70bf75163763e11b34` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:31.059541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466277c9-2ec9-4b80-94b7-691b30d2aac5 None admin] RBAC: Authorizing `identity:get_project(project_id=c0935b3a5a454c70bf75163763e11b34)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:31.061050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466277c9-2ec9-4b80-94b7-691b30d2aac5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:31.063227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 212/438] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:30 2022] GET /identity/v3/projects/c0935b3a5a454c70bf75163763e11b34 => generated 298 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:31.063610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466277c9-2ec9-4b80-94b7-691b30d2aac5 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:31.114937 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41386b09-ef9c-4bba-bd4d-651ffae141a6 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:31.154017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c50f7d2-956b-4e48-83d5-0f7968637c9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:31.154714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c50f7d2-956b-4e48-83d5-0f7968637c9c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:31.155015 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c50f7d2-956b-4e48-83d5-0f7968637c9c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:31.155292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c50f7d2-956b-4e48-83d5-0f7968637c9c None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:31.156145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c50f7d2-956b-4e48-83d5-0f7968637c9c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:31.156462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c50f7d2-956b-4e48-83d5-0f7968637c9c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:31.158109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c50f7d2-956b-4e48-83d5-0f7968637c9c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:31.159675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:31.169958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 227/439] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:31.170762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c50f7d2-956b-4e48-83d5-0f7968637c9c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:31.192252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466277c9-2ec9-4b80-94b7-691b30d2aac5 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:31.205766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:31.247472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2a5e3b-539a-4608-bcb6-0e364980e1eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:31.248930 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a5e3b-539a-4608-bcb6-0e364980e1eb None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:31.249349 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a5e3b-539a-4608-bcb6-0e364980e1eb None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:31.249815 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a5e3b-539a-4608-bcb6-0e364980e1eb None admin] PATH_INFO: `/v3/groups/41df40ce995b4b4380a5d89505417606` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:31.252385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2a5e3b-539a-4608-bcb6-0e364980e1eb None admin] RBAC: Authorizing `identity:get_group(group_id=41df40ce995b4b4380a5d89505417606)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:31.255074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2a5e3b-539a-4608-bcb6-0e364980e1eb None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:31.257872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 213/440] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:31 2022] GET /identity/v3/groups/41df40ce995b4b4380a5d89505417606 => generated 226 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:31.259054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69911279-3865-4e64-b053-85ce9146839f None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:31.259335 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69911279-3865-4e64-b053-85ce9146839f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:31.259631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69911279-3865-4e64-b053-85ce9146839f None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:31.260911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 214/441] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:31.261628 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69911279-3865-4e64-b053-85ce9146839f None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:31.300850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c50f7d2-956b-4e48-83d5-0f7968637c9c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:31.338050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f717f7-ab68-4260-a961-eb06b28120eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:31.338720 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f717f7-ab68-4260-a961-eb06b28120eb None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:31.339812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f717f7-ab68-4260-a961-eb06b28120eb None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:31.339970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f717f7-ab68-4260-a961-eb06b28120eb None admin] PATH_INFO: `/v3/groups/41df40ce995b4b4380a5d89505417606` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:31.341674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f717f7-ab68-4260-a961-eb06b28120eb None admin] RBAC: Authorizing `identity:get_group(group_id=41df40ce995b4b4380a5d89505417606)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:31.344077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f717f7-ab68-4260-a961-eb06b28120eb None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:31.347155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 228/442] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:31 2022] GET /identity/v3/groups/41df40ce995b4b4380a5d89505417606 => generated 226 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:31.348324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f717f7-ab68-4260-a961-eb06b28120eb None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:31.389726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69911279-3865-4e64-b053-85ce9146839f None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:31.430532 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1656b5b6-a212-4058-8713-b5fba7fca340 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:31.431769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1656b5b6-a212-4058-8713-b5fba7fca340 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:31.432218 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1656b5b6-a212-4058-8713-b5fba7fca340 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:31.432576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1656b5b6-a212-4058-8713-b5fba7fca340 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:31.453335 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:31.460804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:31.475978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f717f7-ab68-4260-a961-eb06b28120eb None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:31.523648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c60028-cba9-4aa4-a342-cb592638f26a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:31.524440 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c60028-cba9-4aa4-a342-cb592638f26a None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:31.524740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c60028-cba9-4aa4-a342-cb592638f26a None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:31.525163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c60028-cba9-4aa4-a342-cb592638f26a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:31.525605 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1656b5b6-a212-4058-8713-b5fba7fca340 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:31.526092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c60028-cba9-4aa4-a342-cb592638f26a None admin] RBAC: Adding query filter params (group.id=41df40ce995b4b4380a5d89505417606, role.id=ac68e1820c604122981fd5c4e2ebc447, scope.project.id=2e832f037cf845379232686bfc78f98d) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:31.526404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c60028-cba9-4aa4-a342-cb592638f26a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:31.527528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1656b5b6-a212-4058-8713-b5fba7fca340 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:31.528267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c60028-cba9-4aa4-a342-cb592638f26a None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:31.535582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 229/443] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Jul 31 00:54:31 2022] GET /identity/v3/role_assignments?group.id=41df40ce995b4b4380a5d89505417606&scope.project.id=2e832f037cf845379232686bfc78f98d&role.id=ac68e1820c604122981fd5c4e2ebc447 => generated 611 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:31.537407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e512c1a7-4c38-4625-a631-e3dbc7a41454 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:31.537777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e512c1a7-4c38-4625-a631-e3dbc7a41454 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:31.538057 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e512c1a7-4c38-4625-a631-e3dbc7a41454 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:31.581795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-e512c1a7-4c38-4625-a631-e3dbc7a41454 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:31.583988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e512c1a7-4c38-4625-a631-e3dbc7a41454 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:31.660194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 215/444] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:31.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1656b5b6-a212-4058-8713-b5fba7fca340 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:31.697201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 230/445] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:31.699228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e512c1a7-4c38-4625-a631-e3dbc7a41454 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:31.717390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:31.800407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1656b5b6-a212-4058-8713-b5fba7fca340 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:31.815429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:31.837886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e512c1a7-4c38-4625-a631-e3dbc7a41454 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:31.844840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ccd908-c458-4beb-af29-f819b5fa64c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:31.846439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ccd908-c458-4beb-af29-f819b5fa64c2 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:31.846936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ccd908-c458-4beb-af29-f819b5fa64c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:31.847451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ccd908-c458-4beb-af29-f819b5fa64c2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:31.848527 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ccd908-c458-4beb-af29-f819b5fa64c2 None admin] RBAC: Adding query filter params (group.id=41df40ce995b4b4380a5d89505417606, role.id=0a2f5ca94533412088499eaccd7b2e21, scope.project.id=c0935b3a5a454c70bf75163763e11b34) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:31.848822 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ccd908-c458-4beb-af29-f819b5fa64c2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:31.850277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ccd908-c458-4beb-af29-f819b5fa64c2 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:31.856883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 216/446] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Jul 31 00:54:31 2022] GET /identity/v3/role_assignments?group.id=41df40ce995b4b4380a5d89505417606&scope.project.id=c0935b3a5a454c70bf75163763e11b34&role.id=0a2f5ca94533412088499eaccd7b2e21 => generated 611 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:31.857519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ccd908-c458-4beb-af29-f819b5fa64c2 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:31.891666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5e6009-4354-4280-9457-257ee861d65f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:31.892862 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5e6009-4354-4280-9457-257ee861d65f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:31.893140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5e6009-4354-4280-9457-257ee861d65f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:31.893409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5e6009-4354-4280-9457-257ee861d65f None admin] PATH_INFO: `/v3/projects/18e627c596bb4c99b66fda211d0e4785` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:31.895013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5e6009-4354-4280-9457-257ee861d65f None admin] RBAC: Authorizing `identity:get_project(project_id=18e627c596bb4c99b66fda211d0e4785)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:31.896359 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5e6009-4354-4280-9457-257ee861d65f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:31.902994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 231/447] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:31 2022] GET /identity/v3/projects/18e627c596bb4c99b66fda211d0e4785 => generated 344 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:31.903807 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5e6009-4354-4280-9457-257ee861d65f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:32.021837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ccd908-c458-4beb-af29-f819b5fa64c2 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:32.032409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:32.062726 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5e6009-4354-4280-9457-257ee861d65f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:32.083777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68762c6-caef-4356-b242-347749fd326a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:32.084440 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68762c6-caef-4356-b242-347749fd326a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:32.084730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68762c6-caef-4356-b242-347749fd326a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:32.085011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68762c6-caef-4356-b242-347749fd326a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:32.100215 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68762c6-caef-4356-b242-347749fd326a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-e68762c6-caef-4356-b242-347749fd326a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:32.105708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:32.106748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:32.107705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 217/448] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:54:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:32.115330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68762c6-caef-4356-b242-347749fd326a None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:32.123048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd065dcb-4a2e-48b3-a968-5da6f787df8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:32.123738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd065dcb-4a2e-48b3-a968-5da6f787df8b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:32.124016 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd065dcb-4a2e-48b3-a968-5da6f787df8b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:32.124285 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd065dcb-4a2e-48b3-a968-5da6f787df8b None admin] PATH_INFO: `/v3/groups/8e44f262cddd4b8a9abb0018814b900c` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:32.125483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd065dcb-4a2e-48b3-a968-5da6f787df8b None admin] RBAC: Authorizing `identity:get_group(group_id=8e44f262cddd4b8a9abb0018814b900c)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:32.127228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd065dcb-4a2e-48b3-a968-5da6f787df8b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:32.128944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 232/449] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:31 2022] GET /identity/v3/groups/8e44f262cddd4b8a9abb0018814b900c => generated 229 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:32.135556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd065dcb-4a2e-48b3-a968-5da6f787df8b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:32.161939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:32.268150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68762c6-caef-4356-b242-347749fd326a None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:32.277861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd065dcb-4a2e-48b3-a968-5da6f787df8b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:32.322302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf14c4f8-ff3a-4911-8d04-d6579ce39f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:32.322645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf14c4f8-ff3a-4911-8d04-d6579ce39f80 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:32.322927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf14c4f8-ff3a-4911-8d04-d6579ce39f80 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:32.323233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf14c4f8-ff3a-4911-8d04-d6579ce39f80 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:32.323890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf14c4f8-ff3a-4911-8d04-d6579ce39f80 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:32.324195 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf14c4f8-ff3a-4911-8d04-d6579ce39f80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:32.326029 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf14c4f8-ff3a-4911-8d04-d6579ce39f80 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:32.332876 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b91581-527a-457f-8d5b-2c08b982f73f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:32.334562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b91581-527a-457f-8d5b-2c08b982f73f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:32.334562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b91581-527a-457f-8d5b-2c08b982f73f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:32.334995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b91581-527a-457f-8d5b-2c08b982f73f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:32.336094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 218/450] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:32.340503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b91581-527a-457f-8d5b-2c08b982f73f None admin] RBAC: Adding query filter params (group.id=8e44f262cddd4b8a9abb0018814b900c, role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=18e627c596bb4c99b66fda211d0e4785) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:32.340714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b91581-527a-457f-8d5b-2c08b982f73f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:32.342385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b91581-527a-457f-8d5b-2c08b982f73f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:32.343061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf14c4f8-ff3a-4911-8d04-d6579ce39f80 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:32.349986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 233/451] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Jul 31 00:54:32 2022] GET /identity/v3/role_assignments?group.id=8e44f262cddd4b8a9abb0018814b900c&scope.project.id=18e627c596bb4c99b66fda211d0e4785&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 611 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:32.388696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:32.510738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf14c4f8-ff3a-4911-8d04-d6579ce39f80 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:32.570420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb57c382-e71c-4601-a26a-02bb5bcaf3ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:32.571186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb57c382-e71c-4601-a26a-02bb5bcaf3ce None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:32.571696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb57c382-e71c-4601-a26a-02bb5bcaf3ce None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:32.572040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb57c382-e71c-4601-a26a-02bb5bcaf3ce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:32.669394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb57c382-e71c-4601-a26a-02bb5bcaf3ce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:32.670875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb57c382-e71c-4601-a26a-02bb5bcaf3ce None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:32.700677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:32.790945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 219/452] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 31 00:54:32.797399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b91581-527a-457f-8d5b-2c08b982f73f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:32.894447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:32.922648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b91581-527a-457f-8d5b-2c08b982f73f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:32.964072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5330c58f-14bd-4600-9b94-b79341118e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:32.964719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5330c58f-14bd-4600-9b94-b79341118e84 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:32.964999 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5330c58f-14bd-4600-9b94-b79341118e84 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:32.965270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5330c58f-14bd-4600-9b94-b79341118e84 None admin] PATH_INFO: `/v3/projects/2e832f037cf845379232686bfc78f98d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:32.966526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5330c58f-14bd-4600-9b94-b79341118e84 None admin] RBAC: Authorizing `identity:get_project(project_id=2e832f037cf845379232686bfc78f98d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:32.968011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5330c58f-14bd-4600-9b94-b79341118e84 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:32.969704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 234/453] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:54:32 2022] GET /identity/v3/projects/2e832f037cf845379232686bfc78f98d => generated 302 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:32.975513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb57c382-e71c-4601-a26a-02bb5bcaf3ce None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:33.090693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb57c382-e71c-4601-a26a-02bb5bcaf3ce None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:33.123169 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:33.134138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2861ffca-4ab9-40ca-ad33-5b3428afdfb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:33.134904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2861ffca-4ab9-40ca-ad33-5b3428afdfb9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:33.135184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2861ffca-4ab9-40ca-ad33-5b3428afdfb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:33.135587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2861ffca-4ab9-40ca-ad33-5b3428afdfb9 None admin] PATH_INFO: `/v3/groups/41df40ce995b4b4380a5d89505417606` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:33.137117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2861ffca-4ab9-40ca-ad33-5b3428afdfb9 None admin] RBAC: Authorizing `identity:get_group(group_id=41df40ce995b4b4380a5d89505417606)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:33.138794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2861ffca-4ab9-40ca-ad33-5b3428afdfb9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:33.140770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 220/454] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Jul 31 00:54:32 2022] GET /identity/v3/groups/41df40ce995b4b4380a5d89505417606 => generated 226 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:33.147988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5330c58f-14bd-4600-9b94-b79341118e84 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:33.266624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5330c58f-14bd-4600-9b94-b79341118e84 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:33.306209 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378182db-65e7-489d-9af2-336c787aa13d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:33.306871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378182db-65e7-489d-9af2-336c787aa13d None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:33.307215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378182db-65e7-489d-9af2-336c787aa13d None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:33.307543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378182db-65e7-489d-9af2-336c787aa13d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:33.308365 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378182db-65e7-489d-9af2-336c787aa13d None admin] RBAC: Adding query filter params (group.id=41df40ce995b4b4380a5d89505417606, role.id=0a2f5ca94533412088499eaccd7b2e21, scope.project.id=2e832f037cf845379232686bfc78f98d) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:33.308747 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378182db-65e7-489d-9af2-336c787aa13d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:33.310378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378182db-65e7-489d-9af2-336c787aa13d None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:33.317673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 235/455] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Jul 31 00:54:33 2022] GET /identity/v3/role_assignments?group.id=41df40ce995b4b4380a5d89505417606&scope.project.id=2e832f037cf845379232686bfc78f98d&role.id=0a2f5ca94533412088499eaccd7b2e21 => generated 611 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:36.337603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4324bb35-419a-4f15-a0d0-a992bf95b9b3 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:36.338320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4324bb35-419a-4f15-a0d0-a992bf95b9b3 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:36.338320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4324bb35-419a-4f15-a0d0-a992bf95b9b3 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:36.339912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 221/456] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:36.346398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ae435c-61fb-47ab-ba53-81c56bf3c4a8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:36.347601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ae435c-61fb-47ab-ba53-81c56bf3c4a8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:36.348217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ae435c-61fb-47ab-ba53-81c56bf3c4a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:36.378602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b0ae435c-61fb-47ab-ba53-81c56bf3c4a8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:36.380610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b0ae435c-61fb-47ab-ba53-81c56bf3c4a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:36.399564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:36.515448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 236/457] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:36.558027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4324bb35-419a-4f15-a0d0-a992bf95b9b3 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:36.764656 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4324bb35-419a-4f15-a0d0-a992bf95b9b3 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:36.829397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9ad3c5-0ac0-4e9a-be47-f6a59123c739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:36.830458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9ad3c5-0ac0-4e9a-be47-f6a59123c739 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:36.830936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9ad3c5-0ac0-4e9a-be47-f6a59123c739 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:36.831436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9ad3c5-0ac0-4e9a-be47-f6a59123c739 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:36.839811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c734c8ad-3013-4faa-849b-e59fd49886df None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:36.840304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c734c8ad-3013-4faa-849b-e59fd49886df None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:36.840787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c734c8ad-3013-4faa-849b-e59fd49886df None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:36.843396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 237/458] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:36.848952 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a16f65-cfdc-429f-814c-364183c2d7b9 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:36.849234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a16f65-cfdc-429f-814c-364183c2d7b9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:36.849761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a16f65-cfdc-429f-814c-364183c2d7b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:36.870539 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-e3a16f65-cfdc-429f-814c-364183c2d7b9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:36.872312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e3a16f65-cfdc-429f-814c-364183c2d7b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:36.873142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9ad3c5-0ac0-4e9a-be47-f6a59123c739 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:36.875957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9ad3c5-0ac0-4e9a-be47-f6a59123c739 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-3f9ad3c5-0ac0-4e9a-be47-f6a59123c739 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:36.914567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:36.915622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:36.916433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:36.916433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:36.916433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:36.916433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:36.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 222/459] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 367 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:36.931080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb6086b-cd60-4928-92d8-880e9285d14e None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:36.931402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb6086b-cd60-4928-92d8-880e9285d14e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:36.931795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb6086b-cd60-4928-92d8-880e9285d14e None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:36.933506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 223/460] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:36.934273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb6086b-cd60-4928-92d8-880e9285d14e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:36.974129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 238/461] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:36.984203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c1d3d5-5a25-4fe0-8c79-ac290129c9e4 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:36.984930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c1d3d5-5a25-4fe0-8c79-ac290129c9e4 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:36.985402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c1d3d5-5a25-4fe0-8c79-ac290129c9e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:37.017970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f9c1d3d5-5a25-4fe0-8c79-ac290129c9e4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:37.019495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:37.019776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f9c1d3d5-5a25-4fe0-8c79-ac290129c9e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:37.126098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb6086b-cd60-4928-92d8-880e9285d14e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:37.164073 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 239/462] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:37.164073 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c1d3d5-5a25-4fe0-8c79-ac290129c9e4 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:37.166168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:37.188507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2724e64-1062-441f-95bf-9123eb6d7663 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:37.189732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2724e64-1062-441f-95bf-9123eb6d7663 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:37.190012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2724e64-1062-441f-95bf-9123eb6d7663 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:37.190361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2724e64-1062-441f-95bf-9123eb6d7663 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:37.191085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2724e64-1062-441f-95bf-9123eb6d7663 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:37.191433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2724e64-1062-441f-95bf-9123eb6d7663 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:37.192950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2724e64-1062-441f-95bf-9123eb6d7663 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:37.213065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 224/463] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:37.217427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94b15ef-049f-4648-baf9-2490c44fb860 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:37.217427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94b15ef-049f-4648-baf9-2490c44fb860 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:37.217427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94b15ef-049f-4648-baf9-2490c44fb860 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:37.221415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 225/464] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:37 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:37.221801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94b15ef-049f-4648-baf9-2490c44fb860 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:37.316071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c1d3d5-5a25-4fe0-8c79-ac290129c9e4 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:37.349428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94b15ef-049f-4648-baf9-2490c44fb860 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:37.362467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:37.371052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838f4d32-29a0-4b87-9441-ddae5de5f859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:37.372123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838f4d32-29a0-4b87-9441-ddae5de5f859 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:37.372587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838f4d32-29a0-4b87-9441-ddae5de5f859 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:37.373037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838f4d32-29a0-4b87-9441-ddae5de5f859 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:37.390801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838f4d32-29a0-4b87-9441-ddae5de5f859 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:37.393138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838f4d32-29a0-4b87-9441-ddae5de5f859 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:37.412422 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944d7f19-c110-4da6-b60a-ed4b767f226b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:37.413408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944d7f19-c110-4da6-b60a-ed4b767f226b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:37.413858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944d7f19-c110-4da6-b60a-ed4b767f226b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:37.414309 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944d7f19-c110-4da6-b60a-ed4b767f226b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-838f4d32-29a0-4b87-9441-ddae5de5f859 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:37.419741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:37.421463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:37.424600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:37.424600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:37.424600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:37.424600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:37.430413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 240/465] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:37.431654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838f4d32-29a0-4b87-9441-ddae5de5f859 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:37.435459 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944d7f19-c110-4da6-b60a-ed4b767f226b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:37.438011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944d7f19-c110-4da6-b60a-ed4b767f226b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-944d7f19-c110-4da6-b60a-ed4b767f226b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:37.462394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:37.463312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:37.469642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:37.469642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:37.469642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:37.469642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:37.469642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 226/466] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:37.469642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0de7972-1fad-4e2c-8bd4-50ed0b418fa1 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:37.469642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0de7972-1fad-4e2c-8bd4-50ed0b418fa1 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:37.469642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0de7972-1fad-4e2c-8bd4-50ed0b418fa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:37.494327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b0de7972-1fad-4e2c-8bd4-50ed0b418fa1 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:37.496208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b0de7972-1fad-4e2c-8bd4-50ed0b418fa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:37.528764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:37.601963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 227/467] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:37.602782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0de7972-1fad-4e2c-8bd4-50ed0b418fa1 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:37.613329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838f4d32-29a0-4b87-9441-ddae5de5f859 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:37.659301 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fed3afe-841a-41cc-a465-9d6e5ea0d9f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:37.660034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fed3afe-841a-41cc-a465-9d6e5ea0d9f1 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:37.660372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fed3afe-841a-41cc-a465-9d6e5ea0d9f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:37.660792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fed3afe-841a-41cc-a465-9d6e5ea0d9f1 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:37.661439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fed3afe-841a-41cc-a465-9d6e5ea0d9f1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:37.662925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fed3afe-841a-41cc-a465-9d6e5ea0d9f1 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:37.688732 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 241/468] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:54:37 2022] POST /identity/v3/users => generated 248 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:54:37.689777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fed3afe-841a-41cc-a465-9d6e5ea0d9f1 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:37.771015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:37.786191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0de7972-1fad-4e2c-8bd4-50ed0b418fa1 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:37.836725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddaf5ff-a289-499f-b777-e89464889d02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:37.837974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddaf5ff-a289-499f-b777-e89464889d02 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:37.838468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddaf5ff-a289-499f-b777-e89464889d02 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:37.838946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddaf5ff-a289-499f-b777-e89464889d02 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:37.840398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddaf5ff-a289-499f-b777-e89464889d02 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:37.841177 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddaf5ff-a289-499f-b777-e89464889d02 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:37.842813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddaf5ff-a289-499f-b777-e89464889d02 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:37.866508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 228/469] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:37.868251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddaf5ff-a289-499f-b777-e89464889d02 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:37.878616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fed3afe-841a-41cc-a465-9d6e5ea0d9f1 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:37.901930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:37.939296 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d26544-7dfa-421c-9a24-dcff356159b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:37.940142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d26544-7dfa-421c-9a24-dcff356159b7 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:37.940420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d26544-7dfa-421c-9a24-dcff356159b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:37.940760 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d26544-7dfa-421c-9a24-dcff356159b7 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:37.941463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d26544-7dfa-421c-9a24-dcff356159b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:37.941769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d26544-7dfa-421c-9a24-dcff356159b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:37.943308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d26544-7dfa-421c-9a24-dcff356159b7 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:37.960957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 242/470] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:37.961549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d26544-7dfa-421c-9a24-dcff356159b7 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:38.025409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddaf5ff-a289-499f-b777-e89464889d02 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:38.067071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c5398b-f00e-41ee-a904-4d5e11f39a67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:38.068015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c5398b-f00e-41ee-a904-4d5e11f39a67 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:38.068400 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c5398b-f00e-41ee-a904-4d5e11f39a67 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:38.068780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c5398b-f00e-41ee-a904-4d5e11f39a67 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:38.083955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c5398b-f00e-41ee-a904-4d5e11f39a67 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:38.085465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c5398b-f00e-41ee-a904-4d5e11f39a67 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:38.104644 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d26544-7dfa-421c-9a24-dcff356159b7 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-42c5398b-f00e-41ee-a904-4d5e11f39a67 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:38.104958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:38.105801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:38.106624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:38.106624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:38.106624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:38.106624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:38.106624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 229/471] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:38.107269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c5398b-f00e-41ee-a904-4d5e11f39a67 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:38.110070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:38.157199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa17681-63b4-4687-b449-9c88d629bee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:38.158243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa17681-63b4-4687-b449-9c88d629bee3 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:38.158625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa17681-63b4-4687-b449-9c88d629bee3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:38.159255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa17681-63b4-4687-b449-9c88d629bee3 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:38.160500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa17681-63b4-4687-b449-9c88d629bee3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:38.162152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa17681-63b4-4687-b449-9c88d629bee3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:38.196185 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 243/472] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:54:37 2022] POST /identity/v3/users => generated 246 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:54:38.199283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230e92ed-56fa-4207-b0d7-a52586a83f0b None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:38.199894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230e92ed-56fa-4207-b0d7-a52586a83f0b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:38.200748 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230e92ed-56fa-4207-b0d7-a52586a83f0b None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:38.203438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 244/473] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:38 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:38.205210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230e92ed-56fa-4207-b0d7-a52586a83f0b None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:38.248457 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:38.266019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c5398b-f00e-41ee-a904-4d5e11f39a67 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:38.311434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f34b06d-2de2-48ee-9dd1-1d1e651083af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:38.312375 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f34b06d-2de2-48ee-9dd1-1d1e651083af None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:38.313041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f34b06d-2de2-48ee-9dd1-1d1e651083af None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:38.313161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f34b06d-2de2-48ee-9dd1-1d1e651083af None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:38.314581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f34b06d-2de2-48ee-9dd1-1d1e651083af None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:38.316458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f34b06d-2de2-48ee-9dd1-1d1e651083af None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:38.353528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 230/474] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:54:38 2022] POST /identity/v3/users => generated 249 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:54:38.353528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3b01b8-6162-40da-b50b-5bdf602cada1 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:38.353528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3b01b8-6162-40da-b50b-5bdf602cada1 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:38.353528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3b01b8-6162-40da-b50b-5bdf602cada1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:38.375309 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6a3b01b8-6162-40da-b50b-5bdf602cada1 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:38.377571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6a3b01b8-6162-40da-b50b-5bdf602cada1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:38.384339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230e92ed-56fa-4207-b0d7-a52586a83f0b None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:38.428401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21146a4b-4257-4b5d-a900-0379dbc181c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:38.429257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21146a4b-4257-4b5d-a900-0379dbc181c8 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:38.429682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21146a4b-4257-4b5d-a900-0379dbc181c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:38.430088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21146a4b-4257-4b5d-a900-0379dbc181c8 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:38.430948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21146a4b-4257-4b5d-a900-0379dbc181c8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:38.431440 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21146a4b-4257-4b5d-a900-0379dbc181c8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:38.433113 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21146a4b-4257-4b5d-a900-0379dbc181c8 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:38.462745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 245/475] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:38.473003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21146a4b-4257-4b5d-a900-0379dbc181c8 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:38.488273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:38.517884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 231/476] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:38.545637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3b01b8-6162-40da-b50b-5bdf602cada1 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:38.634551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:38.648650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21146a4b-4257-4b5d-a900-0379dbc181c8 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:38.701733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a526b179-0ec2-4d6c-b700-243243f4bfa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:38.702713 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a526b179-0ec2-4d6c-b700-243243f4bfa8 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:38.703430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a526b179-0ec2-4d6c-b700-243243f4bfa8 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:38.706656 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a526b179-0ec2-4d6c-b700-243243f4bfa8 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:38.707661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a526b179-0ec2-4d6c-b700-243243f4bfa8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:38.709259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a526b179-0ec2-4d6c-b700-243243f4bfa8 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:38.721986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3b01b8-6162-40da-b50b-5bdf602cada1 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:38.736257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 246/477] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:54:38 2022] POST /identity/v3/users => generated 248 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:54:38.765404 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d09026-8ccb-4c9d-998c-636173ca0425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:38.766044 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d09026-8ccb-4c9d-998c-636173ca0425 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:38.766329 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d09026-8ccb-4c9d-998c-636173ca0425 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:38.766690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d09026-8ccb-4c9d-998c-636173ca0425 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:38.767410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d09026-8ccb-4c9d-998c-636173ca0425 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:38.768992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d09026-8ccb-4c9d-998c-636173ca0425 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:38.778402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.sql.core [None req-95d09026-8ccb-4c9d-998c-636173ca0425 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 31 00:54:38.778402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 31 00:54:38.778402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [parameters: {'id': '77d4607ad8824958a9309f3d5a555cb7', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 31 00:54:38.778402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80208) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-95d09026-8ccb-4c9d-998c-636173ca0425 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application result = self._query(query) Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application conn.query(q) Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application result.read() Jul 31 00:54:38.796962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return ref.to_dict() Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 31 00:54:38.798485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application next(self.gen) Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application yield resource Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application next(self.gen) Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self.session.rollback() Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self.force_reraise() Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise self.value Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application session.commit() Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 31 00:54:38.800103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._prepare_impl() Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self.session.flush() Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._flush(objects) Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application compat.raise_( Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application flush_context.execute() Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rec.execute(self) Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 31 00:54:38.802444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 31 00:54:38.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application result = self._query(query) Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application conn.query(q) Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application result.read() Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 31 00:54:38.806497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [parameters: {'id': '77d4607ad8824958a9309f3d5a555cb7', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:38.808415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 31 00:54:38.813338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:38.816099 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 232/478] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Jul 31 00:54:38 2022] POST /identity/v3/roles => generated 144 bytes in 256 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:38.816235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a526b179-0ec2-4d6c-b700-243243f4bfa8 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:38.939978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a526b179-0ec2-4d6c-b700-243243f4bfa8 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:38.981149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8736e836-abdf-428d-9a38-c9586abcc537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:38.982169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8736e836-abdf-428d-9a38-c9586abcc537 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:38.982673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8736e836-abdf-428d-9a38-c9586abcc537 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:38.983143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8736e836-abdf-428d-9a38-c9586abcc537 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:38.997708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8736e836-abdf-428d-9a38-c9586abcc537 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-8736e836-abdf-428d-9a38-c9586abcc537 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:39.004093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:39.005072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:39.005850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:39.005850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:39.005850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:39.005850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:39.005850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:39.005850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:39.005850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:39.005850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:39.005850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:39.005850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:39.005850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 31 00:54:39.005850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:39.008784 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 247/479] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:54:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:39.014732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d09026-8ccb-4c9d-998c-636173ca0425 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:39.089077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:39.169808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d09026-8ccb-4c9d-998c-636173ca0425 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:39.219077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184d8ebc-81b0-4044-b5a0-97139d2bcbe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:39.220686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184d8ebc-81b0-4044-b5a0-97139d2bcbe1 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:39.221174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184d8ebc-81b0-4044-b5a0-97139d2bcbe1 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:39.221636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184d8ebc-81b0-4044-b5a0-97139d2bcbe1 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:39.222577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184d8ebc-81b0-4044-b5a0-97139d2bcbe1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:39.223083 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184d8ebc-81b0-4044-b5a0-97139d2bcbe1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:39.224720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184d8ebc-81b0-4044-b5a0-97139d2bcbe1 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:39.234715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 233/480] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:40.053239 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc2ff35-8387-406c-af0f-170adc9e14ca None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:40.054290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc2ff35-8387-406c-af0f-170adc9e14ca None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:40.054765 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc2ff35-8387-406c-af0f-170adc9e14ca None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:40.056639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 248/481] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:40.067374 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc731a31-853e-45af-81f5-b8444521d033 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:40.067864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc731a31-853e-45af-81f5-b8444521d033 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:40.067936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc731a31-853e-45af-81f5-b8444521d033 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:40.091051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-cc731a31-853e-45af-81f5-b8444521d033 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:40.092304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-cc731a31-853e-45af-81f5-b8444521d033 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:40.190624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 234/482] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:54:40.229198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc2ff35-8387-406c-af0f-170adc9e14ca None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:40.233223 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:40.367671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc2ff35-8387-406c-af0f-170adc9e14ca None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:40.402617 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67e7b36-d76f-4c82-8bb2-e473e2b8ca05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:40.403257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67e7b36-d76f-4c82-8bb2-e473e2b8ca05 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:40.403565 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67e7b36-d76f-4c82-8bb2-e473e2b8ca05 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:40.403836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67e7b36-d76f-4c82-8bb2-e473e2b8ca05 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:40.415068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67e7b36-d76f-4c82-8bb2-e473e2b8ca05 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-c67e7b36-d76f-4c82-8bb2-e473e2b8ca05 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:40.420679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:40.421550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:40.422320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 249/483] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:40.428505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc731a31-853e-45af-81f5-b8444521d033 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:40.488550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:40.559627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc731a31-853e-45af-81f5-b8444521d033 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:40.597869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3705a368-6c6a-48e9-b823-36ad6aa7bcbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:40.598485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3705a368-6c6a-48e9-b823-36ad6aa7bcbc None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:40.598762 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3705a368-6c6a-48e9-b823-36ad6aa7bcbc None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:40.599035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3705a368-6c6a-48e9-b823-36ad6aa7bcbc None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:40.599780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3705a368-6c6a-48e9-b823-36ad6aa7bcbc None admin] RBAC: Adding query filter params (name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:40.600081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3705a368-6c6a-48e9-b823-36ad6aa7bcbc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:40.601462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3705a368-6c6a-48e9-b823-36ad6aa7bcbc None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:40.610440 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 235/484] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:54:40.616047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67e7b36-d76f-4c82-8bb2-e473e2b8ca05 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:40.695887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:40.743039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67e7b36-d76f-4c82-8bb2-e473e2b8ca05 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:40.794895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d16864-99b0-4fe6-8288-1ee8e8fe81cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:40.795568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d16864-99b0-4fe6-8288-1ee8e8fe81cb None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:40.795903 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d16864-99b0-4fe6-8288-1ee8e8fe81cb None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:40.796183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d16864-99b0-4fe6-8288-1ee8e8fe81cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:40.807621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d16864-99b0-4fe6-8288-1ee8e8fe81cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:40.809079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d16864-99b0-4fe6-8288-1ee8e8fe81cb None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-97d16864-99b0-4fe6-8288-1ee8e8fe81cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:40.831282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:40.832274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:40.832274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:40.832274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:40.832274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:40.832274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 250/485] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:40.837134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3705a368-6c6a-48e9-b823-36ad6aa7bcbc None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:40.998220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05b8639-87be-4979-b350-56fee5c0862b None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:40.998530 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05b8639-87be-4979-b350-56fee5c0862b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:40.998790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05b8639-87be-4979-b350-56fee5c0862b None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.000426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 251/486] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:41.005768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7ab1d6-48b5-4baa-9576-427599086cd2 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:41.006030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7ab1d6-48b5-4baa-9576-427599086cd2 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:41.006313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7ab1d6-48b5-4baa-9576-427599086cd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.037325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-fa7ab1d6-48b5-4baa-9576-427599086cd2 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:41.038520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-fa7ab1d6-48b5-4baa-9576-427599086cd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:41.053259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3705a368-6c6a-48e9-b823-36ad6aa7bcbc None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:41.062233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:41.125312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca7f315-e1cd-4f7e-80ba-d09ce0128240 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:41.126134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca7f315-e1cd-4f7e-80ba-d09ce0128240 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:41.126315 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca7f315-e1cd-4f7e-80ba-d09ce0128240 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:41.126495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca7f315-e1cd-4f7e-80ba-d09ce0128240 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.127199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca7f315-e1cd-4f7e-80ba-d09ce0128240 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:41.127526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca7f315-e1cd-4f7e-80ba-d09ce0128240 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:41.128912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca7f315-e1cd-4f7e-80ba-d09ce0128240 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:41.162282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 236/487] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:41.173701 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca7f315-e1cd-4f7e-80ba-d09ce0128240 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:41.190471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 252/488] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:54:41.238220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7ab1d6-48b5-4baa-9576-427599086cd2 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:41.287312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca7f315-e1cd-4f7e-80ba-d09ce0128240 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:41.336514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6b19ad-18c4-425e-801b-f4b498df9264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:41.337173 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6b19ad-18c4-425e-801b-f4b498df9264 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:41.337366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:41.337517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6b19ad-18c4-425e-801b-f4b498df9264 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:41.337703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6b19ad-18c4-425e-801b-f4b498df9264 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.354724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6b19ad-18c4-425e-801b-f4b498df9264 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:41.356210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6b19ad-18c4-425e-801b-f4b498df9264 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-eb6b19ad-18c4-425e-801b-f4b498df9264 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:41.377403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:41.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:41.381277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:41.381277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:41.381277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:41.381277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:41.381277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:41.381277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:41.381277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:41.381277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:41.381277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 31 00:54:41.381277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:41.381277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 237/489] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:41 2022] GET /identity/v3/users/glance => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:41.381970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e45dd5-14e5-40db-ae42-54bac691edec None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:41.381970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e45dd5-14e5-40db-ae42-54bac691edec None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:41.382156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e45dd5-14e5-40db-ae42-54bac691edec None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.384814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 238/490] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:41.384814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e45dd5-14e5-40db-ae42-54bac691edec None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:41.418988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:41.420820 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7ab1d6-48b5-4baa-9576-427599086cd2 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:41.460395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2626e5a9-2300-489e-be70-432ec939b4ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:41.461231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2626e5a9-2300-489e-be70-432ec939b4ce None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:41.461575 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2626e5a9-2300-489e-be70-432ec939b4ce None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:41.461938 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2626e5a9-2300-489e-be70-432ec939b4ce None admin] PATH_INFO: `/v3/roles/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.473462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2626e5a9-2300-489e-be70-432ec939b4ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-2626e5a9-2300-489e-be70-432ec939b4ce None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:41.480320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:41.481401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:41.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:41.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:41.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:41.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:41.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:41.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:41.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:41.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:41.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:41.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:41.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:41.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:41.483336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 253/491] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:41.484345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e3ea1f-19c8-41bc-b6d4-11b30579ab73 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:41.484631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e3ea1f-19c8-41bc-b6d4-11b30579ab73 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:41.484904 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e3ea1f-19c8-41bc-b6d4-11b30579ab73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.503871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b6e3ea1f-19c8-41bc-b6d4-11b30579ab73 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:41.505053 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b6e3ea1f-19c8-41bc-b6d4-11b30579ab73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:41.519572 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e45dd5-14e5-40db-ae42-54bac691edec None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:41.565064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abbb927-1df0-476c-b2d6-8fc2e240c813 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:41.565951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abbb927-1df0-476c-b2d6-8fc2e240c813 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:41.566236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abbb927-1df0-476c-b2d6-8fc2e240c813 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:41.566511 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abbb927-1df0-476c-b2d6-8fc2e240c813 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.567575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abbb927-1df0-476c-b2d6-8fc2e240c813 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:41.568266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abbb927-1df0-476c-b2d6-8fc2e240c813 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:41.569787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abbb927-1df0-476c-b2d6-8fc2e240c813 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:41.584299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 254/492] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:41.585398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17a3a5c-ebcf-41f6-aded-ce0c6fdab6ec None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:41.585679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17a3a5c-ebcf-41f6-aded-ce0c6fdab6ec None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:41.585950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17a3a5c-ebcf-41f6-aded-ce0c6fdab6ec None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.587191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 255/493] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:41 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:41.587895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17a3a5c-ebcf-41f6-aded-ce0c6fdab6ec None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:41.620158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 239/494] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Jul 31 00:54:41 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:41.621933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af24e885-b0c0-4279-81c9-e0ad7a32f477 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:41.622296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af24e885-b0c0-4279-81c9-e0ad7a32f477 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:41.622647 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af24e885-b0c0-4279-81c9-e0ad7a32f477 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.640349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:41.641329 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-af24e885-b0c0-4279-81c9-e0ad7a32f477 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:41.642383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-af24e885-b0c0-4279-81c9-e0ad7a32f477 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:41.712509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 240/495] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:41.713637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af24e885-b0c0-4279-81c9-e0ad7a32f477 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:41.724341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17a3a5c-ebcf-41f6-aded-ce0c6fdab6ec None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:41.737701 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:41.763445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c06b56f-74eb-4f4d-8992-61ead03c1d70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:41.764058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c06b56f-74eb-4f4d-8992-61ead03c1d70 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:41.764769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c06b56f-74eb-4f4d-8992-61ead03c1d70 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:41.765049 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c06b56f-74eb-4f4d-8992-61ead03c1d70 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.765763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c06b56f-74eb-4f4d-8992-61ead03c1d70 None admin] RBAC: Adding query filter params (name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:41.766118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c06b56f-74eb-4f4d-8992-61ead03c1d70 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:41.768228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c06b56f-74eb-4f4d-8992-61ead03c1d70 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:41.781004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 256/496] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:41.781004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c06b56f-74eb-4f4d-8992-61ead03c1d70 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:41.835481 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af24e885-b0c0-4279-81c9-e0ad7a32f477 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:41.872569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b294503d-d861-4455-8e50-af7ae0524822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:41.873208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b294503d-d861-4455-8e50-af7ae0524822 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:41.873488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b294503d-d861-4455-8e50-af7ae0524822 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:41.873968 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b294503d-d861-4455-8e50-af7ae0524822 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.884876 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b294503d-d861-4455-8e50-af7ae0524822 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-b294503d-d861-4455-8e50-af7ae0524822 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:41.889804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:41.890705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:41.891534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 241/497] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:41.892749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b294503d-d861-4455-8e50-af7ae0524822 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:41.900560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:41.906241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c06b56f-74eb-4f4d-8992-61ead03c1d70 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:41.944042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a568f24-339b-424d-839f-375e494f5f30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:41.944775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a568f24-339b-424d-839f-375e494f5f30 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:41.945053 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a568f24-339b-424d-839f-375e494f5f30 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:41.945326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a568f24-339b-424d-839f-375e494f5f30 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:41.974209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:42.002128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b294503d-d861-4455-8e50-af7ae0524822 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:42.021815 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a568f24-339b-424d-839f-375e494f5f30 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:42.023279 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a568f24-339b-424d-839f-375e494f5f30 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:42.038209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412cdb40-047e-472c-b73a-c281676220be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:42.038939 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412cdb40-047e-472c-b73a-c281676220be None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:42.039148 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412cdb40-047e-472c-b73a-c281676220be None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:42.039409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412cdb40-047e-472c-b73a-c281676220be None admin] PATH_INFO: `/v3/roles/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:42.052675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412cdb40-047e-472c-b73a-c281676220be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-412cdb40-047e-472c-b73a-c281676220be None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:42.058128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:42.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:42.060161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 242/498] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:42.064483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412cdb40-047e-472c-b73a-c281676220be None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:42.143886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:42.153908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 257/499] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:42.155027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a568f24-339b-424d-839f-375e494f5f30 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:42.184473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412cdb40-047e-472c-b73a-c281676220be None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:42.228064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874a9c04-156b-4360-8953-0a6a83e1fc92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:42.228708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874a9c04-156b-4360-8953-0a6a83e1fc92 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:42.228985 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874a9c04-156b-4360-8953-0a6a83e1fc92 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:42.229254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874a9c04-156b-4360-8953-0a6a83e1fc92 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:42.231396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:42.240397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874a9c04-156b-4360-8953-0a6a83e1fc92 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:42.241853 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874a9c04-156b-4360-8953-0a6a83e1fc92 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-874a9c04-156b-4360-8953-0a6a83e1fc92 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:42.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:42.257247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:42.258058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:42.258058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:42.258058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:42.258058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:42.258058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 243/500] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:42.258316 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874a9c04-156b-4360-8953-0a6a83e1fc92 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:42.281461 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a568f24-339b-424d-839f-375e494f5f30 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:42.315569 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e214b54-4d3d-4f0d-b19e-73accf49aff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:42.316276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e214b54-4d3d-4f0d-b19e-73accf49aff5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:42.316557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e214b54-4d3d-4f0d-b19e-73accf49aff5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:42.316832 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e214b54-4d3d-4f0d-b19e-73accf49aff5 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:42.317537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e214b54-4d3d-4f0d-b19e-73accf49aff5 None admin] RBAC: Adding query filter params (name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:42.317867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e214b54-4d3d-4f0d-b19e-73accf49aff5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:42.319231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e214b54-4d3d-4f0d-b19e-73accf49aff5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:42.327125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 258/501] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:42.327942 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e214b54-4d3d-4f0d-b19e-73accf49aff5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:42.364004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874a9c04-156b-4360-8953-0a6a83e1fc92 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:42.388613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:42.400567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68c1a30-5eb5-4229-9124-276e3661fa03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:42.401093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68c1a30-5eb5-4229-9124-276e3661fa03 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:42.401690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68c1a30-5eb5-4229-9124-276e3661fa03 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:42.401690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68c1a30-5eb5-4229-9124-276e3661fa03 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:42.402337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68c1a30-5eb5-4229-9124-276e3661fa03 None admin] RBAC: Adding query filter params (name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:42.402635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68c1a30-5eb5-4229-9124-276e3661fa03 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:42.404041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68c1a30-5eb5-4229-9124-276e3661fa03 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:42.412647 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 244/502] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:42.413850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d43974-2adb-41f0-b42d-8306cbeb3ad7 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:42.414132 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d43974-2adb-41f0-b42d-8306cbeb3ad7 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:42.414404 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d43974-2adb-41f0-b42d-8306cbeb3ad7 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:42.415789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 245/503] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:42 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:42.416294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d43974-2adb-41f0-b42d-8306cbeb3ad7 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:42.434995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e214b54-4d3d-4f0d-b19e-73accf49aff5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:42.466324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:42.471742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422711b2-924f-47c6-861b-4b30275f60b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:42.472381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422711b2-924f-47c6-861b-4b30275f60b5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:42.472657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422711b2-924f-47c6-861b-4b30275f60b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:42.472919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422711b2-924f-47c6-861b-4b30275f60b5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:42.483581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422711b2-924f-47c6-861b-4b30275f60b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:42.485033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422711b2-924f-47c6-861b-4b30275f60b5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-422711b2-924f-47c6-861b-4b30275f60b5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:42.499020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:42.500038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:42.500892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:42.500892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:42.500892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:42.500892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:42.500892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 259/504] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:42.501185 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422711b2-924f-47c6-861b-4b30275f60b5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:42.521466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d43974-2adb-41f0-b42d-8306cbeb3ad7 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:42.559259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6d2d78-2bc9-4d69-81f4-a8585fa60a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:42.559926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6d2d78-2bc9-4d69-81f4-a8585fa60a7d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:42.560208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6d2d78-2bc9-4d69-81f4-a8585fa60a7d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:42.560478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6d2d78-2bc9-4d69-81f4-a8585fa60a7d None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:42.561193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6d2d78-2bc9-4d69-81f4-a8585fa60a7d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:42.561498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6d2d78-2bc9-4d69-81f4-a8585fa60a7d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:42.562957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6d2d78-2bc9-4d69-81f4-a8585fa60a7d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:42.576737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 246/505] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:42.577607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6d2d78-2bc9-4d69-81f4-a8585fa60a7d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:42.608902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422711b2-924f-47c6-861b-4b30275f60b5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:42.623779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:42.644333 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94814fc1-a58d-4328-a6ef-7c4323731c2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:42.644976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94814fc1-a58d-4328-a6ef-7c4323731c2e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:42.645256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94814fc1-a58d-4328-a6ef-7c4323731c2e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:42.645532 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94814fc1-a58d-4328-a6ef-7c4323731c2e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:42.656683 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94814fc1-a58d-4328-a6ef-7c4323731c2e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:42.659558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94814fc1-a58d-4328-a6ef-7c4323731c2e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-94814fc1-a58d-4328-a6ef-7c4323731c2e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:42.674212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:42.675079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:42.675950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:42.675950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:42.675950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:42.675950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:42.679501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 260/506] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:42.680933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dfbf1c-0573-4c23-aa13-342d32cd534e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:42.681212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dfbf1c-0573-4c23-aa13-342d32cd534e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:42.681484 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dfbf1c-0573-4c23-aa13-342d32cd534e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:42.681596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6d2d78-2bc9-4d69-81f4-a8585fa60a7d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:42.693094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:42.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-01dfbf1c-0573-4c23-aa13-342d32cd534e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:42.704125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-01dfbf1c-0573-4c23-aa13-342d32cd534e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:42.718467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e313481-62ac-44dc-b253-079057977ce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:42.719106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e313481-62ac-44dc-b253-079057977ce4 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:42.719411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e313481-62ac-44dc-b253-079057977ce4 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:42.720223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e313481-62ac-44dc-b253-079057977ce4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:42.730230 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e313481-62ac-44dc-b253-079057977ce4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:42.731716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e313481-62ac-44dc-b253-079057977ce4 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-1e313481-62ac-44dc-b253-079057977ce4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:42.746812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:42.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:42.748556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:42.748556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:42.748556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:42.748556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:42.748556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 247/507] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:42.748819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e313481-62ac-44dc-b253-079057977ce4 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:42.775839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 261/508] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:42.776682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01dfbf1c-0573-4c23-aa13-342d32cd534e None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:42.859616 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e313481-62ac-44dc-b253-079057977ce4 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:42.870964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:42.879165 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01dfbf1c-0573-4c23-aa13-342d32cd534e None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:42.900404 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f758f27a-f4d2-44f9-b748-054972631efa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:42.901023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f758f27a-f4d2-44f9-b748-054972631efa None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:42.901307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f758f27a-f4d2-44f9-b748-054972631efa None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:42.901583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f758f27a-f4d2-44f9-b748-054972631efa None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:42.902372 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f758f27a-f4d2-44f9-b748-054972631efa None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:42.902684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f758f27a-f4d2-44f9-b748-054972631efa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:42.904111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f758f27a-f4d2-44f9-b748-054972631efa None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:42.914454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46e763a-6898-4828-abe1-de1adaddf1ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:42.915127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46e763a-6898-4828-abe1-de1adaddf1ce None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:42.915536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46e763a-6898-4828-abe1-de1adaddf1ce None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:42.915911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46e763a-6898-4828-abe1-de1adaddf1ce None admin] PATH_INFO: `/v3/users/nova` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:42.916723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 248/509] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:42.917288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f758f27a-f4d2-44f9-b748-054972631efa None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:42.928686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46e763a-6898-4828-abe1-de1adaddf1ce None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:42.930132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46e763a-6898-4828-abe1-de1adaddf1ce None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:42.943445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-e46e763a-6898-4828-abe1-de1adaddf1ce None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:42.943774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:42.945574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:42.945574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:42.945574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:42.945574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:42.945574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:42.945574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:42.945574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:42.945574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:42.945574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:54:42.945574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:42.945574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 262/510] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Jul 31 00:54:42 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:42.946311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46e763a-6898-4828-abe1-de1adaddf1ce None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:43.023948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f758f27a-f4d2-44f9-b748-054972631efa None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:43.047400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46e763a-6898-4828-abe1-de1adaddf1ce None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:43.058434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a90f98-4f4d-45f4-a2c6-ffaab6d6c6eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:43.059046 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a90f98-4f4d-45f4-a2c6-ffaab6d6c6eb None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:43.059330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a90f98-4f4d-45f4-a2c6-ffaab6d6c6eb None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:43.059633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a90f98-4f4d-45f4-a2c6-ffaab6d6c6eb None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:43.060329 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a90f98-4f4d-45f4-a2c6-ffaab6d6c6eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:43.060635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a90f98-4f4d-45f4-a2c6-ffaab6d6c6eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:43.062009 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a90f98-4f4d-45f4-a2c6-ffaab6d6c6eb None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:43.074234 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 249/511] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:43.075047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a90f98-4f4d-45f4-a2c6-ffaab6d6c6eb None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:43.083160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f947a786-b920-428f-ab67-37f398785be5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:43.083822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f947a786-b920-428f-ab67-37f398785be5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:43.084103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f947a786-b920-428f-ab67-37f398785be5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:43.084378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f947a786-b920-428f-ab67-37f398785be5 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:43.085068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f947a786-b920-428f-ab67-37f398785be5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:43.085371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f947a786-b920-428f-ab67-37f398785be5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:43.086752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f947a786-b920-428f-ab67-37f398785be5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:43.098823 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 263/512] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:43.099668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f947a786-b920-428f-ab67-37f398785be5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:43.115476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:43.163052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:43.185134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a90f98-4f4d-45f4-a2c6-ffaab6d6c6eb None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:43.209367 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f947a786-b920-428f-ab67-37f398785be5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:43.220210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53efafe-ce44-40d8-ab6f-6db2dcffc958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:43.220802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53efafe-ce44-40d8-ab6f-6db2dcffc958 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:43.221109 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53efafe-ce44-40d8-ab6f-6db2dcffc958 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:43.221356 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53efafe-ce44-40d8-ab6f-6db2dcffc958 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:43.232169 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53efafe-ce44-40d8-ab6f-6db2dcffc958 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:43.233647 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53efafe-ce44-40d8-ab6f-6db2dcffc958 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:43.246831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170910c0-e646-4f81-a17f-13d79dea41cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:43.247526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170910c0-e646-4f81-a17f-13d79dea41cd None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:43.247810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170910c0-e646-4f81-a17f-13d79dea41cd None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:43.248084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170910c0-e646-4f81-a17f-13d79dea41cd None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-c53efafe-ce44-40d8-ab6f-6db2dcffc958 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:43.249699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:43.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:43.252358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:43.252358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:43.252358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:43.252358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:43.257155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 250/513] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:43.257322 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170910c0-e646-4f81-a17f-13d79dea41cd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:43.258085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53efafe-ce44-40d8-ab6f-6db2dcffc958 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:43.258657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170910c0-e646-4f81-a17f-13d79dea41cd None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-170910c0-e646-4f81-a17f-13d79dea41cd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:54:43.272899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:43.273856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:43.274874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:54:43.274874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:54:43.274874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:54:43.274874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:54:43.274874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:54:43.274874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:54:43.274874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:43.274874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:43.274874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 264/514] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:43.275548 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170910c0-e646-4f81-a17f-13d79dea41cd None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:43.356679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:43.366490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53efafe-ce44-40d8-ab6f-6db2dcffc958 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:43.382628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170910c0-e646-4f81-a17f-13d79dea41cd None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:43.395509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:43.400964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a26581-c445-4a89-a6aa-9c02693deb3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:43.401573 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a26581-c445-4a89-a6aa-9c02693deb3a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:43.401853 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a26581-c445-4a89-a6aa-9c02693deb3a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:43.402126 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a26581-c445-4a89-a6aa-9c02693deb3a None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:43.402941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a26581-c445-4a89-a6aa-9c02693deb3a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:43.403242 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a26581-c445-4a89-a6aa-9c02693deb3a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:43.404708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a26581-c445-4a89-a6aa-9c02693deb3a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:43.416443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d901bd44-ea0f-42c5-aef0-b462a0777b81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:43.417087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d901bd44-ea0f-42c5-aef0-b462a0777b81 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:43.417439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d901bd44-ea0f-42c5-aef0-b462a0777b81 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:43.417717 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d901bd44-ea0f-42c5-aef0-b462a0777b81 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:43.422740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 251/515] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Jul 31 00:54:43 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:43.424158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a26581-c445-4a89-a6aa-9c02693deb3a None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:43.425851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d901bd44-ea0f-42c5-aef0-b462a0777b81 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:43.427614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d901bd44-ea0f-42c5-aef0-b462a0777b81 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-d901bd44-ea0f-42c5-aef0-b462a0777b81 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:43.441366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:43.442249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:43.443105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:43.443105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:43.443105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:43.443105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:43.443105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:43.443105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:43.443105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:43.443105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:43.443105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 31 00:54:43.443105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:43.443105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 265/516] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:43 2022] GET /identity/v3/users/neutron => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:43.443687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d901bd44-ea0f-42c5-aef0-b462a0777b81 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:43.532735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a26581-c445-4a89-a6aa-9c02693deb3a None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:43.545507 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d901bd44-ea0f-42c5-aef0-b462a0777b81 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:43.570126 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a125ef5f-f670-4c63-92e0-d8892607ac0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:43.570757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a125ef5f-f670-4c63-92e0-d8892607ac0d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:43.571036 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a125ef5f-f670-4c63-92e0-d8892607ac0d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:43.571727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a125ef5f-f670-4c63-92e0-d8892607ac0d None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:43.578446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a125ef5f-f670-4c63-92e0-d8892607ac0d None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:43.579948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a125ef5f-f670-4c63-92e0-d8892607ac0d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:43.581642 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7bd5f6-149a-4a69-b040-c6d6b5d0f77c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:43.582264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7bd5f6-149a-4a69-b040-c6d6b5d0f77c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:43.582544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7bd5f6-149a-4a69-b040-c6d6b5d0f77c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:43.582817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7bd5f6-149a-4a69-b040-c6d6b5d0f77c None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:43.583546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7bd5f6-149a-4a69-b040-c6d6b5d0f77c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:43.583852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7bd5f6-149a-4a69-b040-c6d6b5d0f77c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:43.585226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7bd5f6-149a-4a69-b040-c6d6b5d0f77c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:43.586678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-a125ef5f-f670-4c63-92e0-d8892607ac0d None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:43.591498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:43.592483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:43.593346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:43.593346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:43.593346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:43.593346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:43.593346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:43.593346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:43.593346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:43.593346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:43.593346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 31 00:54:43.593346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:43.593346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 252/517] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:43 2022] GET /identity/v3/users/cinder => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:43.593921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a125ef5f-f670-4c63-92e0-d8892607ac0d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:43.602569 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 266/518] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:43.603430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7bd5f6-149a-4a69-b040-c6d6b5d0f77c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:43.649483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:43.700820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a125ef5f-f670-4c63-92e0-d8892607ac0d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:43.703346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7bd5f6-149a-4a69-b040-c6d6b5d0f77c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:43.736031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66658900-a763-44b4-a576-42d9b7baa128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:43.736644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66658900-a763-44b4-a576-42d9b7baa128 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:43.736945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66658900-a763-44b4-a576-42d9b7baa128 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:43.737256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66658900-a763-44b4-a576-42d9b7baa128 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:43.737733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c2ff1a-2027-4c45-a1d5-a43c5f06ffb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:43.737955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66658900-a763-44b4-a576-42d9b7baa128 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:43.738265 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66658900-a763-44b4-a576-42d9b7baa128 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:43.738350 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c2ff1a-2027-4c45-a1d5-a43c5f06ffb3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:43.738603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c2ff1a-2027-4c45-a1d5-a43c5f06ffb3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:43.738870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c2ff1a-2027-4c45-a1d5-a43c5f06ffb3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:43.739748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66658900-a763-44b4-a576-42d9b7baa128 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:43.757228 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 253/519] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:54:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:43.758054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66658900-a763-44b4-a576-42d9b7baa128 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:43.805058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:43.813112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c2ff1a-2027-4c45-a1d5-a43c5f06ffb3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:43.814615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c2ff1a-2027-4c45-a1d5-a43c5f06ffb3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:43.860718 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66658900-a763-44b4-a576-42d9b7baa128 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:43.872435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:43.897471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c194c-36f2-4dd3-aca6-ca3740bbfa24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:43.898101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c194c-36f2-4dd3-aca6-ca3740bbfa24 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:43.898387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c194c-36f2-4dd3-aca6-ca3740bbfa24 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:43.898659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c194c-36f2-4dd3-aca6-ca3740bbfa24 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:43.899483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24c194c-36f2-4dd3-aca6-ca3740bbfa24 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:43.899795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24c194c-36f2-4dd3-aca6-ca3740bbfa24 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:43.901287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24c194c-36f2-4dd3-aca6-ca3740bbfa24 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:43.916495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 267/520] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:43.917374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c2ff1a-2027-4c45-a1d5-a43c5f06ffb3 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:43.925365 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 254/521] 10.222.0.42 () {56 vars in 1139 bytes} [Sun Jul 31 00:54:43 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:43.926882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c194c-36f2-4dd3-aca6-ca3740bbfa24 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:44.026475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:44.034605 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c2ff1a-2027-4c45-a1d5-a43c5f06ffb3 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:44.046661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c194c-36f2-4dd3-aca6-ca3740bbfa24 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:44.071981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ccf842-caae-488b-9e7c-cdb6b192838d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:44.072779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ccf842-caae-488b-9e7c-cdb6b192838d None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:44.073172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ccf842-caae-488b-9e7c-cdb6b192838d None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:44.073583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ccf842-caae-488b-9e7c-cdb6b192838d None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:44.074984 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ccf842-caae-488b-9e7c-cdb6b192838d None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:44.075514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ccf842-caae-488b-9e7c-cdb6b192838d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:44.077840 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ccf842-caae-488b-9e7c-cdb6b192838d None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:44.089697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d3767-df57-49f2-bbe6-092fe909b229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:44.090795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d3767-df57-49f2-bbe6-092fe909b229 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:44.091337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d3767-df57-49f2-bbe6-092fe909b229 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:44.091982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d3767-df57-49f2-bbe6-092fe909b229 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:44.093154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434d3767-df57-49f2-bbe6-092fe909b229 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:44.095749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434d3767-df57-49f2-bbe6-092fe909b229 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:44.124404 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 255/522] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:54:43 2022] POST /identity/v3/users => generated 247 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:54:44.125301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d3767-df57-49f2-bbe6-092fe909b229 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:44.142735 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 268/523] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Jul 31 00:54:43 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:44.144197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ccf842-caae-488b-9e7c-cdb6b192838d None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:44.182748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:44.282986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d3767-df57-49f2-bbe6-092fe909b229 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:44.289956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ccf842-caae-488b-9e7c-cdb6b192838d None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:44.324307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be52ca9e-e2f2-4dff-a15a-0632a18669a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:44.324945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be52ca9e-e2f2-4dff-a15a-0632a18669a0 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:44.325233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be52ca9e-e2f2-4dff-a15a-0632a18669a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:44.325515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be52ca9e-e2f2-4dff-a15a-0632a18669a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:44.326323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be52ca9e-e2f2-4dff-a15a-0632a18669a0 None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=4160039561434e51a87c6c7a019f2dff) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:44.326632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be52ca9e-e2f2-4dff-a15a-0632a18669a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:44.328223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be52ca9e-e2f2-4dff-a15a-0632a18669a0 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:44.334044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22e20ef-84ca-4001-b9fa-d9946c7627cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:44.334496 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 256/524] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:44 2022] GET /identity/v3/role_assignments?user.id=4160039561434e51a87c6c7a019f2dff&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 259 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:44.335129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22e20ef-84ca-4001-b9fa-d9946c7627cb None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:44.335338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be52ca9e-e2f2-4dff-a15a-0632a18669a0 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:44.335789 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22e20ef-84ca-4001-b9fa-d9946c7627cb None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:44.336069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22e20ef-84ca-4001-b9fa-d9946c7627cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:44.353355 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22e20ef-84ca-4001-b9fa-d9946c7627cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:44.355254 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22e20ef-84ca-4001-b9fa-d9946c7627cb None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-d22e20ef-84ca-4001-b9fa-d9946c7627cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:44.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:44.379317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:44.379317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:44.379317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:44.379317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:44.379317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 269/525] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:44.379317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22e20ef-84ca-4001-b9fa-d9946c7627cb None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:44.384068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:44.477524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:44.496007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be52ca9e-e2f2-4dff-a15a-0632a18669a0 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:44.511176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22e20ef-84ca-4001-b9fa-d9946c7627cb None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:44.551761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a830370d-c8b7-48a8-bde0-38f3da11370f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:44.552393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a830370d-c8b7-48a8-bde0-38f3da11370f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:44.552669 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a830370d-c8b7-48a8-bde0-38f3da11370f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:44.552976 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a830370d-c8b7-48a8-bde0-38f3da11370f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:44.562010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83f1aa0-c4e6-497a-b250-4819cb8252b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:44.563189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83f1aa0-c4e6-497a-b250-4819cb8252b7 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:44.563545 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83f1aa0-c4e6-497a-b250-4819cb8252b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:44.563823 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83f1aa0-c4e6-497a-b250-4819cb8252b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:44.627995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a830370d-c8b7-48a8-bde0-38f3da11370f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:44.629804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a830370d-c8b7-48a8-bde0-38f3da11370f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:44.652537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83f1aa0-c4e6-497a-b250-4819cb8252b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:44.654740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83f1aa0-c4e6-497a-b250-4819cb8252b7 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:44.658846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:44.737579 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:44.742037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 257/526] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:44.744237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a830370d-c8b7-48a8-bde0-38f3da11370f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:44.786268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 270/527] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:44.787203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83f1aa0-c4e6-497a-b250-4819cb8252b7 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:44.869072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a830370d-c8b7-48a8-bde0-38f3da11370f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:44.900708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83f1aa0-c4e6-497a-b250-4819cb8252b7 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:44.903228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:44.912916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48571552-6197-4491-84ac-befd803b508d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:44.913642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48571552-6197-4491-84ac-befd803b508d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:44.913924 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48571552-6197-4491-84ac-befd803b508d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:44.914197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48571552-6197-4491-84ac-befd803b508d None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:44.915000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48571552-6197-4491-84ac-befd803b508d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:44.915306 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48571552-6197-4491-84ac-befd803b508d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:44.916744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48571552-6197-4491-84ac-befd803b508d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:44.929892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 258/528] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:44.930997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48571552-6197-4491-84ac-befd803b508d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:44.940497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9ed310-535a-4dc9-aba5-88bb6cc29419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:44.941247 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9ed310-535a-4dc9-aba5-88bb6cc29419 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:44.941527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9ed310-535a-4dc9-aba5-88bb6cc29419 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:44.941798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9ed310-535a-4dc9-aba5-88bb6cc29419 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:44.952859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9ed310-535a-4dc9-aba5-88bb6cc29419 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:44.954880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9ed310-535a-4dc9-aba5-88bb6cc29419 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-fd9ed310-535a-4dc9-aba5-88bb6cc29419 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:44.971397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:44.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:44.975382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:44.975382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:44.975382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:44.975382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:44.975382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 271/529] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:44.975382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9ed310-535a-4dc9-aba5-88bb6cc29419 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:44.985122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:45.043699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48571552-6197-4491-84ac-befd803b508d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:45.080804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77fce2c-69b5-4eb0-bb10-af3b6cb1b31c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:45.081650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fce2c-69b5-4eb0-bb10-af3b6cb1b31c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:45.081999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fce2c-69b5-4eb0-bb10-af3b6cb1b31c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:45.082341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fce2c-69b5-4eb0-bb10-af3b6cb1b31c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:45.084464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9ed310-535a-4dc9-aba5-88bb6cc29419 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:45.095097 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77fce2c-69b5-4eb0-bb10-af3b6cb1b31c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:45.097005 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77fce2c-69b5-4eb0-bb10-af3b6cb1b31c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-d77fce2c-69b5-4eb0-bb10-af3b6cb1b31c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:45.113731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:45.115703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:45.118198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:45.118198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:45.118198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:45.118198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:45.118198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 259/530] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:45.118198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77fce2c-69b5-4eb0-bb10-af3b6cb1b31c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:45.122313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f2eefd-b7d9-49ea-bc5e-6b3542795119 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:45.122924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f2eefd-b7d9-49ea-bc5e-6b3542795119 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:45.123208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f2eefd-b7d9-49ea-bc5e-6b3542795119 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:45.127374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f2eefd-b7d9-49ea-bc5e-6b3542795119 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:45.131698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f2eefd-b7d9-49ea-bc5e-6b3542795119 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:45.133352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f2eefd-b7d9-49ea-bc5e-6b3542795119 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-76f2eefd-b7d9-49ea-bc5e-6b3542795119 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:54:45.146598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:45.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:45.150573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:54:45.150573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:54:45.150573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:54:45.150573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:54:45.150573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:54:45.150573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:54:45.150573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:45.150573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:45.150573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 272/531] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:45.150573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f2eefd-b7d9-49ea-bc5e-6b3542795119 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:45.170383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:45.226783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:45.229782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77fce2c-69b5-4eb0-bb10-af3b6cb1b31c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:45.264381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f2eefd-b7d9-49ea-bc5e-6b3542795119 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:45.269213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c48ba38-a7b4-4969-a870-bafe05fb1be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:45.269754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c48ba38-a7b4-4969-a870-bafe05fb1be6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:45.270102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c48ba38-a7b4-4969-a870-bafe05fb1be6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:45.270452 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c48ba38-a7b4-4969-a870-bafe05fb1be6 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:45.271219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c48ba38-a7b4-4969-a870-bafe05fb1be6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:45.271632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c48ba38-a7b4-4969-a870-bafe05fb1be6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:45.273094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c48ba38-a7b4-4969-a870-bafe05fb1be6 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:45.288374 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 260/532] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:45.289432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c48ba38-a7b4-4969-a870-bafe05fb1be6 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:45.312683 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cd93bf-0bff-4761-9031-97e70086deb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:45.313663 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cd93bf-0bff-4761-9031-97e70086deb7 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:45.314120 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cd93bf-0bff-4761-9031-97e70086deb7 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:45.314614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cd93bf-0bff-4761-9031-97e70086deb7 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:45.315631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cd93bf-0bff-4761-9031-97e70086deb7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:45.316112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cd93bf-0bff-4761-9031-97e70086deb7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:45.317652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cd93bf-0bff-4761-9031-97e70086deb7 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:45.331743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 273/533] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:45.332856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cd93bf-0bff-4761-9031-97e70086deb7 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:45.404084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c48ba38-a7b4-4969-a870-bafe05fb1be6 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:45.441263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:45.445358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c75b90e-1786-4794-9ab3-d9db6218db54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:45.446242 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c75b90e-1786-4794-9ab3-d9db6218db54 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:45.446753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c75b90e-1786-4794-9ab3-d9db6218db54 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:45.447234 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c75b90e-1786-4794-9ab3-d9db6218db54 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:45.448197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c75b90e-1786-4794-9ab3-d9db6218db54 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:45.448706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c75b90e-1786-4794-9ab3-d9db6218db54 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:45.451287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c75b90e-1786-4794-9ab3-d9db6218db54 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:45.466065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 261/534] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:54:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:45.467278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c75b90e-1786-4794-9ab3-d9db6218db54 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:45.469064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cd93bf-0bff-4761-9031-97e70086deb7 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:45.493424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:45.516321 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae04f1f1-4926-4247-9258-fe5ab636c501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:45.517731 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae04f1f1-4926-4247-9258-fe5ab636c501 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:45.518416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae04f1f1-4926-4247-9258-fe5ab636c501 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:45.518691 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae04f1f1-4926-4247-9258-fe5ab636c501 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:45.535514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae04f1f1-4926-4247-9258-fe5ab636c501 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:45.537890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae04f1f1-4926-4247-9258-fe5ab636c501 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-ae04f1f1-4926-4247-9258-fe5ab636c501 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:54:45.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:45.564847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:45.566267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:54:45.566267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:54:45.566267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:54:45.566267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:54:45.566267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:54:45.566267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:54:45.566267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:45.566267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:45.566267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 274/535] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:45.566267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae04f1f1-4926-4247-9258-fe5ab636c501 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:45.601201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c75b90e-1786-4794-9ab3-d9db6218db54 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:45.643533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da3d16-c9af-4549-89b1-199502dc195c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:45.644188 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da3d16-c9af-4549-89b1-199502dc195c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:45.644467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da3d16-c9af-4549-89b1-199502dc195c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:45.644736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da3d16-c9af-4549-89b1-199502dc195c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:45.657951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53da3d16-c9af-4549-89b1-199502dc195c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:45.661451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53da3d16-c9af-4549-89b1-199502dc195c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-53da3d16-c9af-4549-89b1-199502dc195c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:54:45.677003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:45.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:45.678833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:54:45.678833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:54:45.678833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:54:45.678833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:54:45.678833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:54:45.678833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:54:45.678833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:45.678833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:45.681374 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 262/536] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:45.682448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da3d16-c9af-4549-89b1-199502dc195c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:45.706225 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae04f1f1-4926-4247-9258-fe5ab636c501 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:45.747082 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c110c5-15f9-40ad-9dd8-e1f66fa3ed15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:45.748098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c110c5-15f9-40ad-9dd8-e1f66fa3ed15 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:45.748580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c110c5-15f9-40ad-9dd8-e1f66fa3ed15 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:45.749071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c110c5-15f9-40ad-9dd8-e1f66fa3ed15 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:45.750055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c110c5-15f9-40ad-9dd8-e1f66fa3ed15 None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=5b0bd572ae7a4c7f9e07fe0b9708aa7e) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:45.750646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c110c5-15f9-40ad-9dd8-e1f66fa3ed15 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:45.752455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c110c5-15f9-40ad-9dd8-e1f66fa3ed15 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:45.756093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:45.760324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 275/537] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:45 2022] GET /identity/v3/role_assignments?user.id=5b0bd572ae7a4c7f9e07fe0b9708aa7e&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:45.761489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c110c5-15f9-40ad-9dd8-e1f66fa3ed15 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:45.786174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:45.838257 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da3d16-c9af-4549-89b1-199502dc195c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:45.920597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5090cbc-f803-40bc-beac-7661fb036e24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:45.921756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5090cbc-f803-40bc-beac-7661fb036e24 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:45.922379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5090cbc-f803-40bc-beac-7661fb036e24 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:45.923853 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5090cbc-f803-40bc-beac-7661fb036e24 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:45.926874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5090cbc-f803-40bc-beac-7661fb036e24 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:45.927474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5090cbc-f803-40bc-beac-7661fb036e24 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:45.929155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5090cbc-f803-40bc-beac-7661fb036e24 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:45.957818 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 263/538] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:54:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:45.963638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5090cbc-f803-40bc-beac-7661fb036e24 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:45.971438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c110c5-15f9-40ad-9dd8-e1f66fa3ed15 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:46.048556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb1faf6-7011-4ec2-add8-04a1bb12e1d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:46.049172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb1faf6-7011-4ec2-add8-04a1bb12e1d7 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:46.049445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb1faf6-7011-4ec2-add8-04a1bb12e1d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:46.049707 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb1faf6-7011-4ec2-add8-04a1bb12e1d7 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:46.050728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb1faf6-7011-4ec2-add8-04a1bb12e1d7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:46.051031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb1faf6-7011-4ec2-add8-04a1bb12e1d7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:46.052744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb1faf6-7011-4ec2-add8-04a1bb12e1d7 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:46.073206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 276/539] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:54:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:46.078831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb1faf6-7011-4ec2-add8-04a1bb12e1d7 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:46.123266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5090cbc-f803-40bc-beac-7661fb036e24 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:46.165958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e13abd-a101-4625-b855-42fe768b247d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:46.166967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e13abd-a101-4625-b855-42fe768b247d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:46.167513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e13abd-a101-4625-b855-42fe768b247d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:46.168123 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e13abd-a101-4625-b855-42fe768b247d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:46.169123 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e13abd-a101-4625-b855-42fe768b247d None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=94e868131d33472b9ded8486274a07b9) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:46.169648 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e13abd-a101-4625-b855-42fe768b247d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:46.171425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e13abd-a101-4625-b855-42fe768b247d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:46.179163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 264/540] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:45 2022] GET /identity/v3/role_assignments?user.id=94e868131d33472b9ded8486274a07b9&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 259 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:46.205761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:46.261671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb1faf6-7011-4ec2-add8-04a1bb12e1d7 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:46.308049 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72eda11f-3ef7-49b4-aecb-a45a96845552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:46.308760 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72eda11f-3ef7-49b4-aecb-a45a96845552 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:46.309060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72eda11f-3ef7-49b4-aecb-a45a96845552 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:46.309340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72eda11f-3ef7-49b4-aecb-a45a96845552 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:46.310169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72eda11f-3ef7-49b4-aecb-a45a96845552 None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=9c8338bf95d34b228176b556afd93155) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:46.310475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72eda11f-3ef7-49b4-aecb-a45a96845552 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:46.312181 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72eda11f-3ef7-49b4-aecb-a45a96845552 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:46.320727 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 277/541] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:46 2022] GET /identity/v3/role_assignments?user.id=9c8338bf95d34b228176b556afd93155&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 259 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:46.530606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d76870-93bf-4203-8ab2-80c3ccc66620 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:46.531607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d76870-93bf-4203-8ab2-80c3ccc66620 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:46.532639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d76870-93bf-4203-8ab2-80c3ccc66620 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:46.534253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 265/542] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:46.539883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9185e4-79c6-4306-baa9-438b800c755a None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:46.540392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9185e4-79c6-4306-baa9-438b800c755a None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:46.540468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9185e4-79c6-4306-baa9-438b800c755a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:46.576478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-4d9185e4-79c6-4306-baa9-438b800c755a None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:46.577749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4d9185e4-79c6-4306-baa9-438b800c755a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:46.618992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce504044-5867-4af7-912d-32f12d51968d None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:46.619307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce504044-5867-4af7-912d-32f12d51968d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:46.619706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce504044-5867-4af7-912d-32f12d51968d None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:46.621316 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 266/543] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:46.626664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab43685-bda8-42a7-ad62-4fb9a963b77b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:46.627236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab43685-bda8-42a7-ad62-4fb9a963b77b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:46.627750 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab43685-bda8-42a7-ad62-4fb9a963b77b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:46.643730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:46.660852 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-7ab43685-bda8-42a7-ad62-4fb9a963b77b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:46.662064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7ab43685-bda8-42a7-ad62-4fb9a963b77b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:46.672481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 278/544] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:46.701177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9185e4-79c6-4306-baa9-438b800c755a None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:46.781472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 267/545] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:46.787712 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:46.823817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab43685-bda8-42a7-ad62-4fb9a963b77b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:46.870587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9185e4-79c6-4306-baa9-438b800c755a None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:46.939383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab43685-bda8-42a7-ad62-4fb9a963b77b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:46.978023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d719d0-bcf5-4451-a95a-0ee8e5c5ff3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:46.979008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d719d0-bcf5-4451-a95a-0ee8e5c5ff3a None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:46.980940 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d719d0-bcf5-4451-a95a-0ee8e5c5ff3a None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:46.981424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d719d0-bcf5-4451-a95a-0ee8e5c5ff3a None admin] PATH_INFO: `/v3/roles/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:46.992642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ed77ea-6395-42a8-b413-b477f00faeaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:46.993582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed77ea-6395-42a8-b413-b477f00faeaa None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:46.994036 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed77ea-6395-42a8-b413-b477f00faeaa None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:46.994667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed77ea-6395-42a8-b413-b477f00faeaa None admin] PATH_INFO: `/v3/roles/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:46.998983 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:47.004020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d719d0-bcf5-4451-a95a-0ee8e5c5ff3a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-57d719d0-bcf5-4451-a95a-0ee8e5c5ff3a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:47.009409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:47.010280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:47.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:47.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:47.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:47.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:47.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:47.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:47.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:47.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:47.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:47.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:47.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:47.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:47.019114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 279/546] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:47.023908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d719d0-bcf5-4451-a95a-0ee8e5c5ff3a None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:47.025475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ed77ea-6395-42a8-b413-b477f00faeaa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-c9ed77ea-6395-42a8-b413-b477f00faeaa None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:47.030364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:47.031221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:47.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:47.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:47.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:47.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:47.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:47.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:47.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:47.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:47.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:47.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:47.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:47.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:47.035806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 268/547] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:47.044357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ed77ea-6395-42a8-b413-b477f00faeaa None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:47.151393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d719d0-bcf5-4451-a95a-0ee8e5c5ff3a None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:47.169805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:47.195109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f24ff3-1072-4a68-a22b-d8d4feaf6cd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:47.195830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f24ff3-1072-4a68-a22b-d8d4feaf6cd5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:47.196110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f24ff3-1072-4a68-a22b-d8d4feaf6cd5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:47.196382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f24ff3-1072-4a68-a22b-d8d4feaf6cd5 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:47.197106 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f24ff3-1072-4a68-a22b-d8d4feaf6cd5 None admin] RBAC: Adding query filter params (name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:47.197409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f24ff3-1072-4a68-a22b-d8d4feaf6cd5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:47.199016 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f24ff3-1072-4a68-a22b-d8d4feaf6cd5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:47.209709 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 280/548] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:47.215826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f24ff3-1072-4a68-a22b-d8d4feaf6cd5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:47.298506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ed77ea-6395-42a8-b413-b477f00faeaa None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:47.329659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f24ff3-1072-4a68-a22b-d8d4feaf6cd5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:47.365168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7098976-36ad-431a-b812-dd88f6c630c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:47.365813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7098976-36ad-431a-b812-dd88f6c630c6 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:47.366141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7098976-36ad-431a-b812-dd88f6c630c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:47.366421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7098976-36ad-431a-b812-dd88f6c630c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:47.366821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28bd407-2947-4f96-9d1e-eb806e6821ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:47.367668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28bd407-2947-4f96-9d1e-eb806e6821ec None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:47.368252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28bd407-2947-4f96-9d1e-eb806e6821ec None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:47.368736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28bd407-2947-4f96-9d1e-eb806e6821ec None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:47.369745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28bd407-2947-4f96-9d1e-eb806e6821ec None admin] RBAC: Adding query filter params (name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:47.370245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28bd407-2947-4f96-9d1e-eb806e6821ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:47.371855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28bd407-2947-4f96-9d1e-eb806e6821ec None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:47.378267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7098976-36ad-431a-b812-dd88f6c630c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:47.383005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7098976-36ad-431a-b812-dd88f6c630c6 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:47.389962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 269/549] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 346 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:47.397359 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28bd407-2947-4f96-9d1e-eb806e6821ec None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-f7098976-36ad-431a-b812-dd88f6c630c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:47.399909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:47.400899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:47.401751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:47.401751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:47.401751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:47.401751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:47.401751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 281/550] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:47.406829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7098976-36ad-431a-b812-dd88f6c630c6 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:47.450367 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:47.522542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:47.602395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7098976-36ad-431a-b812-dd88f6c630c6 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:47.608065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28bd407-2947-4f96-9d1e-eb806e6821ec None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:47.683914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775784b3-3e11-4c4a-ac2c-957a3da2c96b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:47.692091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775784b3-3e11-4c4a-ac2c-957a3da2c96b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:47.692431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775784b3-3e11-4c4a-ac2c-957a3da2c96b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:47.692647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775784b3-3e11-4c4a-ac2c-957a3da2c96b None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:47.693404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775784b3-3e11-4c4a-ac2c-957a3da2c96b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:47.693711 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775784b3-3e11-4c4a-ac2c-957a3da2c96b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:47.695144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775784b3-3e11-4c4a-ac2c-957a3da2c96b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:47.698176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffc2bf8-5c06-4891-a1f2-a3775c8d5049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:47.698943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffc2bf8-5c06-4891-a1f2-a3775c8d5049 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:47.699236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffc2bf8-5c06-4891-a1f2-a3775c8d5049 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:47.699548 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffc2bf8-5c06-4891-a1f2-a3775c8d5049 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:47.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffc2bf8-5c06-4891-a1f2-a3775c8d5049 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:47.719385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffc2bf8-5c06-4891-a1f2-a3775c8d5049 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:47.722199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 282/551] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:47.727586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775784b3-3e11-4c4a-ac2c-957a3da2c96b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-5ffc2bf8-5c06-4891-a1f2-a3775c8d5049 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:47.746957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:47.747974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:47.748892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:47.748892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:47.748892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:47.748892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:47.756063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 270/552] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 358 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:47.756943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffc2bf8-5c06-4891-a1f2-a3775c8d5049 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:47.897879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:47.912597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775784b3-3e11-4c4a-ac2c-957a3da2c96b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:47.949494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ee8c54-202a-4619-bb16-25050a1079bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:47.950459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ee8c54-202a-4619-bb16-25050a1079bf None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:47.950826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ee8c54-202a-4619-bb16-25050a1079bf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:47.951265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ee8c54-202a-4619-bb16-25050a1079bf None admin] PATH_INFO: `/v3/users/swift` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:47.960589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ee8c54-202a-4619-bb16-25050a1079bf None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:47.962100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ee8c54-202a-4619-bb16-25050a1079bf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-44ee8c54-202a-4619-bb16-25050a1079bf None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:47.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:47.977061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:47.977963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:47.977963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:47.977963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:47.977963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:47.977963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:47.977963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:47.977963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:47.977963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:47.977963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 31 00:54:47.977963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:47.977963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 283/553] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:54:47 2022] GET /identity/v3/users/swift => generated 83 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:47.984843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ee8c54-202a-4619-bb16-25050a1079bf None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:48.000552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffc2bf8-5c06-4891-a1f2-a3775c8d5049 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:48.015854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:48.080271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced578a3-1491-46d7-8c43-07271cfb0614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:48.081325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced578a3-1491-46d7-8c43-07271cfb0614 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.081811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced578a3-1491-46d7-8c43-07271cfb0614 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.082310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced578a3-1491-46d7-8c43-07271cfb0614 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.083236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced578a3-1491-46d7-8c43-07271cfb0614 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:48.083845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced578a3-1491-46d7-8c43-07271cfb0614 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:48.085471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced578a3-1491-46d7-8c43-07271cfb0614 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:48.105511 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 271/554] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 349 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:48.107447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9058ad42-d6d6-4c2d-9c9d-4e241c248624 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.107986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9058ad42-d6d6-4c2d-9c9d-4e241c248624 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.108491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9058ad42-d6d6-4c2d-9c9d-4e241c248624 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.111007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 272/555] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:48.112076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9058ad42-d6d6-4c2d-9c9d-4e241c248624 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:48.123313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ee8c54-202a-4619-bb16-25050a1079bf None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:48.166967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142ce2db-0306-49cb-8056-e3e6e1f61e8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:48.167663 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ce2db-0306-49cb-8056-e3e6e1f61e8e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.167947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ce2db-0306-49cb-8056-e3e6e1f61e8e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.168224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ce2db-0306-49cb-8056-e3e6e1f61e8e None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.169136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142ce2db-0306-49cb-8056-e3e6e1f61e8e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:48.169450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142ce2db-0306-49cb-8056-e3e6e1f61e8e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:48.171074 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142ce2db-0306-49cb-8056-e3e6e1f61e8e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:48.200169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 284/556] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Jul 31 00:54:47 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:48.200169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b424cd40-e327-4889-a005-380c5c8213c9 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.200169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b424cd40-e327-4889-a005-380c5c8213c9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.200169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b424cd40-e327-4889-a005-380c5c8213c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.212769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:48.222245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b424cd40-e327-4889-a005-380c5c8213c9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:48.223960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b424cd40-e327-4889-a005-380c5c8213c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:48.270493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9058ad42-d6d6-4c2d-9c9d-4e241c248624 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:48.315381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 285/557] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:48.315381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b424cd40-e327-4889-a005-380c5c8213c9 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:48.338418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ab34d5-3538-4467-9c73-fde28e518f1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:48.339326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ab34d5-3538-4467-9c73-fde28e518f1a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.339864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ab34d5-3538-4467-9c73-fde28e518f1a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ab34d5-3538-4467-9c73-fde28e518f1a None admin] PATH_INFO: `/v3/users/glance` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.349435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ab34d5-3538-4467-9c73-fde28e518f1a None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:48.351250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ab34d5-3538-4467-9c73-fde28e518f1a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-82ab34d5-3538-4467-9c73-fde28e518f1a None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:48.364700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:48.366145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:48.367063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:48.367063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:48.367063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:48.367063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:48.367063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:48.367063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:48.367063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:48.367063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:48.367063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 31 00:54:48.367063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:48.374323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 273/558] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:48 2022] GET /identity/v3/users/glance => generated 84 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:48.375761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ab34d5-3538-4467-9c73-fde28e518f1a None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:48.408330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:48.425119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b424cd40-e327-4889-a005-380c5c8213c9 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:48.461683 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7718245c-cc0e-4a6c-93fe-2924a44cd1bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:48.462311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7718245c-cc0e-4a6c-93fe-2924a44cd1bf None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.462597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7718245c-cc0e-4a6c-93fe-2924a44cd1bf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.462871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7718245c-cc0e-4a6c-93fe-2924a44cd1bf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.466831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:48.496430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ab34d5-3538-4467-9c73-fde28e518f1a None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:48.538248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f9c532-684a-41c0-87a9-e802c3f903cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:48.539152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f9c532-684a-41c0-87a9-e802c3f903cb None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.539606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f9c532-684a-41c0-87a9-e802c3f903cb None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.540053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f9c532-684a-41c0-87a9-e802c3f903cb None admin] PATH_INFO: `/v3/roles/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.544284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7718245c-cc0e-4a6c-93fe-2924a44cd1bf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:48.545816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7718245c-cc0e-4a6c-93fe-2924a44cd1bf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:48.554229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f9c532-684a-41c0-87a9-e802c3f903cb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-b6f9c532-684a-41c0-87a9-e802c3f903cb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:48.560249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:48.561445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:48.562377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:48.562377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:48.562377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:48.562377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:48.562377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:48.562377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:48.562377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:48.562377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:48.562377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:48.562377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:48.562377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:48.562377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:48.565307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 274/559] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:48.566571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f9c532-684a-41c0-87a9-e802c3f903cb None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:48.660296 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 286/560] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:48.661176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7718245c-cc0e-4a6c-93fe-2924a44cd1bf None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:48.685651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:48.701022 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f9c532-684a-41c0-87a9-e802c3f903cb None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:48.705324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:48.739512 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6095659-9cf9-43c4-a545-9bf695925ef1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:48.740133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6095659-9cf9-43c4-a545-9bf695925ef1 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.740406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6095659-9cf9-43c4-a545-9bf695925ef1 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.741920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6095659-9cf9-43c4-a545-9bf695925ef1 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.742732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6095659-9cf9-43c4-a545-9bf695925ef1 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:48.743040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6095659-9cf9-43c4-a545-9bf695925ef1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:48.744539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6095659-9cf9-43c4-a545-9bf695925ef1 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:48.763542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 275/561] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Jul 31 00:54:48 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:48.764809 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e89663d-4e26-4e8e-acf3-1bcc02c0df4e None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.765094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e89663d-4e26-4e8e-acf3-1bcc02c0df4e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.765364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e89663d-4e26-4e8e-acf3-1bcc02c0df4e None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.766612 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 276/562] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:48.767310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e89663d-4e26-4e8e-acf3-1bcc02c0df4e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:48.788698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7718245c-cc0e-4a6c-93fe-2924a44cd1bf None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:48.825501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1043f878-6643-465f-bb5c-e132ec734484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:48.826117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1043f878-6643-465f-bb5c-e132ec734484 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.826395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1043f878-6643-465f-bb5c-e132ec734484 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.826667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1043f878-6643-465f-bb5c-e132ec734484 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.827928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1043f878-6643-465f-bb5c-e132ec734484 None admin] RBAC: Adding query filter params (name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:48.827928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1043f878-6643-465f-bb5c-e132ec734484 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:48.829114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1043f878-6643-465f-bb5c-e132ec734484 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:48.841132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 287/563] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:48.842405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd29a5-a300-4138-b2f9-e712a6e55890 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.842695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd29a5-a300-4138-b2f9-e712a6e55890 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.842970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd29a5-a300-4138-b2f9-e712a6e55890 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.848094 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 288/564] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:48.848094 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dd29a5-a300-4138-b2f9-e712a6e55890 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:48.886690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e89663d-4e26-4e8e-acf3-1bcc02c0df4e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:48.929523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999ba68b-b5cc-4f8e-b1ed-5d6ad02fc811 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:48.930538 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999ba68b-b5cc-4f8e-b1ed-5d6ad02fc811 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.930959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999ba68b-b5cc-4f8e-b1ed-5d6ad02fc811 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.931238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999ba68b-b5cc-4f8e-b1ed-5d6ad02fc811 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.948768 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999ba68b-b5cc-4f8e-b1ed-5d6ad02fc811 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:48.950147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999ba68b-b5cc-4f8e-b1ed-5d6ad02fc811 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:48.958818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:48.959940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:48.964376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dd29a5-a300-4138-b2f9-e712a6e55890 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-999ba68b-b5cc-4f8e-b1ed-5d6ad02fc811 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:48.969338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:48.971473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:48.973597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:48.973597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:48.973597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:48.973597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:48.973597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 277/565] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:48.973597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c910b37-379b-4b90-bfaf-8c3170ac63d4 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:48.973597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c910b37-379b-4b90-bfaf-8c3170ac63d4 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:48.973597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c910b37-379b-4b90-bfaf-8c3170ac63d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:48.995209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-8c910b37-379b-4b90-bfaf-8c3170ac63d4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:48.996443 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8c910b37-379b-4b90-bfaf-8c3170ac63d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:49.003383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae7b90c-afc9-4262-91cf-fede83e9b937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:49.004032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae7b90c-afc9-4262-91cf-fede83e9b937 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:49.004304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae7b90c-afc9-4262-91cf-fede83e9b937 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:49.004567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae7b90c-afc9-4262-91cf-fede83e9b937 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:49.017083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae7b90c-afc9-4262-91cf-fede83e9b937 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:49.018567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae7b90c-afc9-4262-91cf-fede83e9b937 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-3ae7b90c-afc9-4262-91cf-fede83e9b937 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:49.041316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:49.042295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:49.043148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:49.043148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:49.043148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:49.043148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:49.043906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 289/566] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:49.044178 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae7b90c-afc9-4262-91cf-fede83e9b937 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:49.080062 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 278/567] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:49.081523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02493dba-6e7d-424b-9636-5606a7e7d3e7 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:49.081806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02493dba-6e7d-424b-9636-5606a7e7d3e7 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:49.082074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02493dba-6e7d-424b-9636-5606a7e7d3e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:49.109342 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-02493dba-6e7d-424b-9636-5606a7e7d3e7 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:49.110565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-02493dba-6e7d-424b-9636-5606a7e7d3e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:49.172020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae7b90c-afc9-4262-91cf-fede83e9b937 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:49.200129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 279/568] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:49.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02493dba-6e7d-424b-9636-5606a7e7d3e7 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:49.210506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083eaab8-8aeb-4775-8827-a4d1f2c707ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:49.211146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083eaab8-8aeb-4775-8827-a4d1f2c707ee None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:49.211450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083eaab8-8aeb-4775-8827-a4d1f2c707ee None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:49.211798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083eaab8-8aeb-4775-8827-a4d1f2c707ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:49.230757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083eaab8-8aeb-4775-8827-a4d1f2c707ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:49.233131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083eaab8-8aeb-4775-8827-a4d1f2c707ee None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-083eaab8-8aeb-4775-8827-a4d1f2c707ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:49.248302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:49.249883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:49.252198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:49.252198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:49.252198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:49.252198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:49.252198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 290/569] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:49.252198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083eaab8-8aeb-4775-8827-a4d1f2c707ee None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:49.254086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:49.289370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:49.345502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02493dba-6e7d-424b-9636-5606a7e7d3e7 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:49.367383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083eaab8-8aeb-4775-8827-a4d1f2c707ee None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:49.392380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955ed3cc-4540-4bde-80c7-4b60c2ecf16f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:49.393135 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ed3cc-4540-4bde-80c7-4b60c2ecf16f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:49.393498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ed3cc-4540-4bde-80c7-4b60c2ecf16f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:49.393960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ed3cc-4540-4bde-80c7-4b60c2ecf16f None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:49.394800 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955ed3cc-4540-4bde-80c7-4b60c2ecf16f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:49.395102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955ed3cc-4540-4bde-80c7-4b60c2ecf16f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:49.397143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955ed3cc-4540-4bde-80c7-4b60c2ecf16f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:49.404832 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e30550-c065-4117-8c2d-9fcaf812ecd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:49.405447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e30550-c065-4117-8c2d-9fcaf812ecd8 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:49.405728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e30550-c065-4117-8c2d-9fcaf812ecd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:49.406001 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e30550-c065-4117-8c2d-9fcaf812ecd8 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:49.406716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e30550-c065-4117-8c2d-9fcaf812ecd8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:49.407586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e30550-c065-4117-8c2d-9fcaf812ecd8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:49.408967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e30550-c065-4117-8c2d-9fcaf812ecd8 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:49.413533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 280/570] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:49.414363 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955ed3cc-4540-4bde-80c7-4b60c2ecf16f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:49.421328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 291/571] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:49.422176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e30550-c065-4117-8c2d-9fcaf812ecd8 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:49.497734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:49.538915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955ed3cc-4540-4bde-80c7-4b60c2ecf16f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:49.541315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e30550-c065-4117-8c2d-9fcaf812ecd8 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:49.562405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:49.583041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1015fb2-9069-4f5f-9289-168bfe501c94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:49.583726 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1015fb2-9069-4f5f-9289-168bfe501c94 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:49.584006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1015fb2-9069-4f5f-9289-168bfe501c94 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:49.584278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1015fb2-9069-4f5f-9289-168bfe501c94 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:49.586061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0545ba40-d796-4dbf-8b07-a67ff7a27710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:49.586749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0545ba40-d796-4dbf-8b07-a67ff7a27710 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:49.587120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0545ba40-d796-4dbf-8b07-a67ff7a27710 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:49.587498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0545ba40-d796-4dbf-8b07-a67ff7a27710 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:49.595696 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1015fb2-9069-4f5f-9289-168bfe501c94 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:49.598669 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0545ba40-d796-4dbf-8b07-a67ff7a27710 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-a1015fb2-9069-4f5f-9289-168bfe501c94 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:49.600666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:49.601557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:49.602357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 292/572] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:49.603111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1015fb2-9069-4f5f-9289-168bfe501c94 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-0545ba40-d796-4dbf-8b07-a67ff7a27710 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:49.604111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:49.605144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 281/573] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:49.611586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0545ba40-d796-4dbf-8b07-a67ff7a27710 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:49.722412 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1015fb2-9069-4f5f-9289-168bfe501c94 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:49.726353 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0545ba40-d796-4dbf-8b07-a67ff7a27710 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:49.747148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:49.762734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a3ecc3-e10c-4e7f-8784-8272d7b05728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:49.762943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22054341-834d-4c37-be5c-b3263d87ffee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:49.763269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a3ecc3-e10c-4e7f-8784-8272d7b05728 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:49.763567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a3ecc3-e10c-4e7f-8784-8272d7b05728 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:49.763677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22054341-834d-4c37-be5c-b3263d87ffee None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:49.763836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a3ecc3-e10c-4e7f-8784-8272d7b05728 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:49.763947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22054341-834d-4c37-be5c-b3263d87ffee None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:49.764177 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22054341-834d-4c37-be5c-b3263d87ffee None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:49.764525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a3ecc3-e10c-4e7f-8784-8272d7b05728 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:49.764824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a3ecc3-e10c-4e7f-8784-8272d7b05728 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:49.766189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a3ecc3-e10c-4e7f-8784-8272d7b05728 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:49.775370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22054341-834d-4c37-be5c-b3263d87ffee None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:49.776840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22054341-834d-4c37-be5c-b3263d87ffee None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:49.780475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 293/574] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:49.781730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a3ecc3-e10c-4e7f-8784-8272d7b05728 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-22054341-834d-4c37-be5c-b3263d87ffee None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:54:49.791216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:49.792283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:49.793146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:54:49.793146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:54:49.793146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:54:49.793146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:54:49.793146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:54:49.793146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:54:49.793146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:49.793146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:49.793146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 282/575] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:49.793955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22054341-834d-4c37-be5c-b3263d87ffee None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:49.840892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:49.896231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a3ecc3-e10c-4e7f-8784-8272d7b05728 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:49.905248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22054341-834d-4c37-be5c-b3263d87ffee None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:49.935369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30be2aa-1fba-4333-bcd8-348fbc143c44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:49.936192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30be2aa-1fba-4333-bcd8-348fbc143c44 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:49.936515 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30be2aa-1fba-4333-bcd8-348fbc143c44 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:49.936825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30be2aa-1fba-4333-bcd8-348fbc143c44 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:49.947550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30be2aa-1fba-4333-bcd8-348fbc143c44 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:49.949443 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e96097-3487-4fa2-9ee4-f5ff8f6ecd40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:49.950018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30be2aa-1fba-4333-bcd8-348fbc143c44 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:49.950395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e96097-3487-4fa2-9ee4-f5ff8f6ecd40 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:49.950862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e96097-3487-4fa2-9ee4-f5ff8f6ecd40 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:49.951310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e96097-3487-4fa2-9ee4-f5ff8f6ecd40 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:49.952544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e96097-3487-4fa2-9ee4-f5ff8f6ecd40 None admin] RBAC: Adding query filter params (name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:49.952898 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e96097-3487-4fa2-9ee4-f5ff8f6ecd40 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:49.954505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e96097-3487-4fa2-9ee4-f5ff8f6ecd40 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-d30be2aa-1fba-4333-bcd8-348fbc143c44 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:54:49.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:49.963467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:49.964342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:54:49.964342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:54:49.964342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:54:49.964342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:54:49.964342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:54:49.964342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:54:49.964342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:49.964342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:49.964342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 294/576] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:49.964836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30be2aa-1fba-4333-bcd8-348fbc143c44 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:49.970738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 283/577] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:49.971688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e96097-3487-4fa2-9ee4-f5ff8f6ecd40 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:50.022384 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:50.079314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:50.082703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e96097-3487-4fa2-9ee4-f5ff8f6ecd40 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:50.084045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30be2aa-1fba-4333-bcd8-348fbc143c44 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:50.117983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b9edae-9914-463e-952a-35787c6ca2f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:50.118393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e146e4-b47c-49d0-b432-f0d2299e08ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:50.118603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b9edae-9914-463e-952a-35787c6ca2f9 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:50.118882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b9edae-9914-463e-952a-35787c6ca2f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:50.119027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e146e4-b47c-49d0-b432-f0d2299e08ed None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:50.119154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b9edae-9914-463e-952a-35787c6ca2f9 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:50.119303 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e146e4-b47c-49d0-b432-f0d2299e08ed None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:50.119601 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e146e4-b47c-49d0-b432-f0d2299e08ed None admin] PATH_INFO: `/v3/users/nova` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:50.119906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b9edae-9914-463e-952a-35787c6ca2f9 None admin] RBAC: Adding query filter params (name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:50.120210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b9edae-9914-463e-952a-35787c6ca2f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:50.121579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b9edae-9914-463e-952a-35787c6ca2f9 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:50.130538 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e146e4-b47c-49d0-b432-f0d2299e08ed None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:50.131590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e146e4-b47c-49d0-b432-f0d2299e08ed None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:50.132238 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 295/578] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:50.133694 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b9edae-9914-463e-952a-35787c6ca2f9 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-60e146e4-b47c-49d0-b432-f0d2299e08ed None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:50.145874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:50.146891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:50.148210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:50.148210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:50.148210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:50.148210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:50.148210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:50.148210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:50.148210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:50.148210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:50.148210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:54:50.148210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:50.148210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 284/579] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Jul 31 00:54:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:50.148774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e146e4-b47c-49d0-b432-f0d2299e08ed None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:50.252688 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b9edae-9914-463e-952a-35787c6ca2f9 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:50.255067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:50.265923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e146e4-b47c-49d0-b432-f0d2299e08ed None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:50.289722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575d3a43-c3d1-4378-afc9-1afede2cdfc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:50.290252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575d3a43-c3d1-4378-afc9-1afede2cdfc5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:50.290533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575d3a43-c3d1-4378-afc9-1afede2cdfc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:50.290793 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575d3a43-c3d1-4378-afc9-1afede2cdfc5 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:50.291564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575d3a43-c3d1-4378-afc9-1afede2cdfc5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:50.291877 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575d3a43-c3d1-4378-afc9-1afede2cdfc5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:50.293298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575d3a43-c3d1-4378-afc9-1afede2cdfc5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:50.305645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f92f5c-add7-4da9-90bd-df1e35c00212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:50.306251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 296/580] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:54:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:50.306532 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f92f5c-add7-4da9-90bd-df1e35c00212 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:50.306532 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f92f5c-add7-4da9-90bd-df1e35c00212 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:50.306770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f92f5c-add7-4da9-90bd-df1e35c00212 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:50.307057 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575d3a43-c3d1-4378-afc9-1afede2cdfc5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:50.307537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f92f5c-add7-4da9-90bd-df1e35c00212 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:50.307915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f92f5c-add7-4da9-90bd-df1e35c00212 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:50.309369 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f92f5c-add7-4da9-90bd-df1e35c00212 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:50.323837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 285/581] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:54:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:50.324755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f92f5c-add7-4da9-90bd-df1e35c00212 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:50.350680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:50.416003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575d3a43-c3d1-4378-afc9-1afede2cdfc5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:50.443148 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f92f5c-add7-4da9-90bd-df1e35c00212 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:50.450870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1042c444-f05a-4372-9c4a-96ffcd4c2579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:50.451537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1042c444-f05a-4372-9c4a-96ffcd4c2579 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:50.451819 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1042c444-f05a-4372-9c4a-96ffcd4c2579 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:50.452089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1042c444-f05a-4372-9c4a-96ffcd4c2579 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:50.463877 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1042c444-f05a-4372-9c4a-96ffcd4c2579 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:50.465370 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1042c444-f05a-4372-9c4a-96ffcd4c2579 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-1042c444-f05a-4372-9c4a-96ffcd4c2579 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:50.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:50.487400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:50.488520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:50.488520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:50.488520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:50.488520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:50.488757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973c3e81-6d82-42f4-adac-424d68d9cd89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:50.488757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973c3e81-6d82-42f4-adac-424d68d9cd89 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:50.488757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973c3e81-6d82-42f4-adac-424d68d9cd89 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:50.488981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 297/582] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:50.489058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973c3e81-6d82-42f4-adac-424d68d9cd89 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:50.489134 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1042c444-f05a-4372-9c4a-96ffcd4c2579 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:50.495765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973c3e81-6d82-42f4-adac-424d68d9cd89 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:50.497251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973c3e81-6d82-42f4-adac-424d68d9cd89 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:50.503884 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-973c3e81-6d82-42f4-adac-424d68d9cd89 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:50.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:50.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:50.517586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:50.517586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:50.517586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:50.517586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:50.517586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 286/583] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:50.517586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973c3e81-6d82-42f4-adac-424d68d9cd89 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:50.601979 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:50.608051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1042c444-f05a-4372-9c4a-96ffcd4c2579 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:50.633003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973c3e81-6d82-42f4-adac-424d68d9cd89 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:50.642574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9746d951-1edd-4637-b238-71c4939bc3b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:50.643179 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9746d951-1edd-4637-b238-71c4939bc3b7 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:50.643489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9746d951-1edd-4637-b238-71c4939bc3b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:50.643760 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9746d951-1edd-4637-b238-71c4939bc3b7 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:50.644548 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9746d951-1edd-4637-b238-71c4939bc3b7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:50.644847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9746d951-1edd-4637-b238-71c4939bc3b7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:50.646279 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9746d951-1edd-4637-b238-71c4939bc3b7 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:50.663937 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 298/584] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Jul 31 00:54:50 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:50.665344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9746d951-1edd-4637-b238-71c4939bc3b7 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:50.678516 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869df195-1e68-4be7-9c4f-dc7f6e0c369b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:50.679198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869df195-1e68-4be7-9c4f-dc7f6e0c369b None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:50.679507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869df195-1e68-4be7-9c4f-dc7f6e0c369b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:50.679783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869df195-1e68-4be7-9c4f-dc7f6e0c369b None admin] PATH_INFO: `/v3/projects/8b41573c9f3146d19a306089d686015c/users/4160039561434e51a87c6c7a019f2dff/roles/26925162cb82467f981eae19826e00c9` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:50.698000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869df195-1e68-4be7-9c4f-dc7f6e0c369b None admin] RBAC: Authorizing `identity:create_grant(project_id=8b41573c9f3146d19a306089d686015c, user_id=4160039561434e51a87c6c7a019f2dff, role_id=26925162cb82467f981eae19826e00c9)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:50.699498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869df195-1e68-4be7-9c4f-dc7f6e0c369b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:50.714098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 287/585] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:54:50 2022] PUT /identity/v3/projects/8b41573c9f3146d19a306089d686015c/users/4160039561434e51a87c6c7a019f2dff/roles/26925162cb82467f981eae19826e00c9 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:50.714588 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869df195-1e68-4be7-9c4f-dc7f6e0c369b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:50.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:50.789387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9746d951-1edd-4637-b238-71c4939bc3b7 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:50.831454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3789b7-253b-49b2-b85e-e9fd1f2de9b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:50.831986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3789b7-253b-49b2-b85e-e9fd1f2de9b4 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:50.832264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3789b7-253b-49b2-b85e-e9fd1f2de9b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:50.832560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3789b7-253b-49b2-b85e-e9fd1f2de9b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:50.833368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3789b7-253b-49b2-b85e-e9fd1f2de9b4 None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=c11f1dd5a6b042c9b864a5861d0f3569) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:50.833678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3789b7-253b-49b2-b85e-e9fd1f2de9b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:50.835156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3789b7-253b-49b2-b85e-e9fd1f2de9b4 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:50.851850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 299/586] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:50 2022] GET /identity/v3/role_assignments?user.id=c11f1dd5a6b042c9b864a5861d0f3569&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 259 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:50.852589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3789b7-253b-49b2-b85e-e9fd1f2de9b4 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:50.887458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869df195-1e68-4be7-9c4f-dc7f6e0c369b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:50.922195 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:50.948477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c4f641-2743-476d-8f98-0e72edde7807 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:50.949493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c4f641-2743-476d-8f98-0e72edde7807 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:50.949774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c4f641-2743-476d-8f98-0e72edde7807 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:50.950046 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c4f641-2743-476d-8f98-0e72edde7807 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:50.951073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c4f641-2743-476d-8f98-0e72edde7807 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:50.951901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c4f641-2743-476d-8f98-0e72edde7807 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:50.953690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c4f641-2743-476d-8f98-0e72edde7807 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:50.971798 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 288/587] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:50.972713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c4f641-2743-476d-8f98-0e72edde7807 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:51.008164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3789b7-253b-49b2-b85e-e9fd1f2de9b4 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:51.054090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17d9ac0-c3a8-4921-82c5-6577b81c3985 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:51.054864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17d9ac0-c3a8-4921-82c5-6577b81c3985 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:51.055269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17d9ac0-c3a8-4921-82c5-6577b81c3985 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:51.055585 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17d9ac0-c3a8-4921-82c5-6577b81c3985 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:51.056444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17d9ac0-c3a8-4921-82c5-6577b81c3985 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:51.056748 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17d9ac0-c3a8-4921-82c5-6577b81c3985 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:51.058951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17d9ac0-c3a8-4921-82c5-6577b81c3985 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:51.067708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:51.074771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 300/588] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:51.075714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17d9ac0-c3a8-4921-82c5-6577b81c3985 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:51.098975 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c4f641-2743-476d-8f98-0e72edde7807 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:51.144351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd103a1c-f09d-4047-a713-358eb253bec6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:51.145060 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd103a1c-f09d-4047-a713-358eb253bec6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:51.145338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd103a1c-f09d-4047-a713-358eb253bec6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:51.145657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd103a1c-f09d-4047-a713-358eb253bec6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:51.161160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd103a1c-f09d-4047-a713-358eb253bec6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:51.163007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd103a1c-f09d-4047-a713-358eb253bec6 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-fd103a1c-f09d-4047-a713-358eb253bec6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:51.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:51.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:51.182508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:51.182508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:51.182508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:51.182508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:51.184520 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 289/589] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:51.185608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd103a1c-f09d-4047-a713-358eb253bec6 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:51.210328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17d9ac0-c3a8-4921-82c5-6577b81c3985 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:51.225896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:51.252629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee4f894-4c52-424b-90a3-bd43d7bb2953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:51.253373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee4f894-4c52-424b-90a3-bd43d7bb2953 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:51.253649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee4f894-4c52-424b-90a3-bd43d7bb2953 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:51.253917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee4f894-4c52-424b-90a3-bd43d7bb2953 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:51.262294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee4f894-4c52-424b-90a3-bd43d7bb2953 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:51.263881 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee4f894-4c52-424b-90a3-bd43d7bb2953 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-dee4f894-4c52-424b-90a3-bd43d7bb2953 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:51.277558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:51.278785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:51.279806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:51.279806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:51.279806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:51.279806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:51.279806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:51.279806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:51.279806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:51.279806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:51.279806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 31 00:54:51.279806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:51.284444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 301/590] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:51 2022] GET /identity/v3/users/cinder => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:51.285022 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee4f894-4c52-424b-90a3-bd43d7bb2953 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:51.304347 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd103a1c-f09d-4047-a713-358eb253bec6 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:51.340824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97bfd9e-4a2d-4c68-baaa-b34440c0382a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:51.341735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97bfd9e-4a2d-4c68-baaa-b34440c0382a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:51.342156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97bfd9e-4a2d-4c68-baaa-b34440c0382a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:51.342593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97bfd9e-4a2d-4c68-baaa-b34440c0382a None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:51.352251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97bfd9e-4a2d-4c68-baaa-b34440c0382a None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:51.353661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97bfd9e-4a2d-4c68-baaa-b34440c0382a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:51.363753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-c97bfd9e-4a2d-4c68-baaa-b34440c0382a None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:51.367814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:51.368745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:51.369654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:51.369654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:51.369654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:51.369654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:51.369654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:51.369654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:51.369654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:51.369654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:51.369654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 31 00:54:51.369654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:51.369654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 290/591] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:51 2022] GET /identity/v3/users/neutron => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:51.370242 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97bfd9e-4a2d-4c68-baaa-b34440c0382a None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:51.404478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee4f894-4c52-424b-90a3-bd43d7bb2953 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:51.446859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30704b8-f6e4-4a78-a525-3b5a2b3a2e80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:51.447548 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30704b8-f6e4-4a78-a525-3b5a2b3a2e80 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:51.447844 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30704b8-f6e4-4a78-a525-3b5a2b3a2e80 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:51.448122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30704b8-f6e4-4a78-a525-3b5a2b3a2e80 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:51.448830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30704b8-f6e4-4a78-a525-3b5a2b3a2e80 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:51.449144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30704b8-f6e4-4a78-a525-3b5a2b3a2e80 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:51.450734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30704b8-f6e4-4a78-a525-3b5a2b3a2e80 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:51.467178 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 302/592] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:51.468302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30704b8-f6e4-4a78-a525-3b5a2b3a2e80 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:51.470786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:51.478565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97bfd9e-4a2d-4c68-baaa-b34440c0382a None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:51.541287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdd46d7-e057-42f9-941e-a6ba8d1aac7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:51.542249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdd46d7-e057-42f9-941e-a6ba8d1aac7d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:51.542678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdd46d7-e057-42f9-941e-a6ba8d1aac7d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:51.543098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdd46d7-e057-42f9-941e-a6ba8d1aac7d None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:51.544152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdd46d7-e057-42f9-941e-a6ba8d1aac7d None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:51.544629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdd46d7-e057-42f9-941e-a6ba8d1aac7d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:51.546327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdd46d7-e057-42f9-941e-a6ba8d1aac7d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:51.567787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 291/593] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Jul 31 00:54:51 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:51.568987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdd46d7-e057-42f9-941e-a6ba8d1aac7d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:51.620055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30704b8-f6e4-4a78-a525-3b5a2b3a2e80 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:51.642336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:51.664462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285425a-7daf-4941-8089-edbf2c6b39bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:51.665263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285425a-7daf-4941-8089-edbf2c6b39bb None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:51.665567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285425a-7daf-4941-8089-edbf2c6b39bb None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:51.665841 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285425a-7daf-4941-8089-edbf2c6b39bb None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:51.666653 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4285425a-7daf-4941-8089-edbf2c6b39bb None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:51.666958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4285425a-7daf-4941-8089-edbf2c6b39bb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:51.669501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4285425a-7daf-4941-8089-edbf2c6b39bb None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:51.692099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 303/594] 10.222.0.42 () {56 vars in 1139 bytes} [Sun Jul 31 00:54:51 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:51.692960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285425a-7daf-4941-8089-edbf2c6b39bb None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:51.711229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdd46d7-e057-42f9-941e-a6ba8d1aac7d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:51.752049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6efdef2-608e-48b6-9c13-bde2515368d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:51.753067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6efdef2-608e-48b6-9c13-bde2515368d2 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:51.753502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6efdef2-608e-48b6-9c13-bde2515368d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:51.753927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6efdef2-608e-48b6-9c13-bde2515368d2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:51.761688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6efdef2-608e-48b6-9c13-bde2515368d2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:51.763454 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6efdef2-608e-48b6-9c13-bde2515368d2 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-c6efdef2-608e-48b6-9c13-bde2515368d2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:54:51.776902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:51.777987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:51.779015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:54:51.779015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:54:51.779015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:54:51.779015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:54:51.779015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:54:51.779015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:54:51.779015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:51.779015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:51.781297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 292/595] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:51.782338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6efdef2-608e-48b6-9c13-bde2515368d2 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:51.798888 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:51.817620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285425a-7daf-4941-8089-edbf2c6b39bb None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:51.855142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd8447e-2549-4795-bea3-66fbeaa7e6d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:51.856060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd8447e-2549-4795-bea3-66fbeaa7e6d8 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:51.856674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd8447e-2549-4795-bea3-66fbeaa7e6d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:51.857106 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd8447e-2549-4795-bea3-66fbeaa7e6d8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:51.871120 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd8447e-2549-4795-bea3-66fbeaa7e6d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:51.872997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd8447e-2549-4795-bea3-66fbeaa7e6d8 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:51.892946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6efdef2-608e-48b6-9c13-bde2515368d2 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-edd8447e-2549-4795-bea3-66fbeaa7e6d8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:51.894140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:51.895189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:51.896176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:51.896176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:51.896176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:51.896176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:51.898297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 304/596] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:51.898505 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd8447e-2549-4795-bea3-66fbeaa7e6d8 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:51.934496 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bff3279-8e5d-40d0-b68c-7e3b0a2a5e74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:51.935074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bff3279-8e5d-40d0-b68c-7e3b0a2a5e74 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:51.935366 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bff3279-8e5d-40d0-b68c-7e3b0a2a5e74 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:51.935643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bff3279-8e5d-40d0-b68c-7e3b0a2a5e74 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:51.938786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:51.951525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bff3279-8e5d-40d0-b68c-7e3b0a2a5e74 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:51.953203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bff3279-8e5d-40d0-b68c-7e3b0a2a5e74 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-6bff3279-8e5d-40d0-b68c-7e3b0a2a5e74 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:51.970830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:51.971832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:51.973846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:51.973846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:51.973846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:51.973846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:51.973846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 293/597] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:51.973846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bff3279-8e5d-40d0-b68c-7e3b0a2a5e74 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:52.051981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd8447e-2549-4795-bea3-66fbeaa7e6d8 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:52.071836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:52.093251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bff3279-8e5d-40d0-b68c-7e3b0a2a5e74 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:52.101287 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdad5f7-1b61-426f-ba0b-0b673361b6ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:52.102584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdad5f7-1b61-426f-ba0b-0b673361b6ef None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:52.103318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdad5f7-1b61-426f-ba0b-0b673361b6ef None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:52.104079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdad5f7-1b61-426f-ba0b-0b673361b6ef None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:52.105854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdad5f7-1b61-426f-ba0b-0b673361b6ef None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:52.106639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdad5f7-1b61-426f-ba0b-0b673361b6ef None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:52.109529 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdad5f7-1b61-426f-ba0b-0b673361b6ef None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:52.124912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 305/598] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:54:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:52.126714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdad5f7-1b61-426f-ba0b-0b673361b6ef None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:52.133632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fa9f61-b546-46bf-a10d-b563941829e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:52.134314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa9f61-b546-46bf-a10d-b563941829e8 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:52.134694 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa9f61-b546-46bf-a10d-b563941829e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:52.134977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa9f61-b546-46bf-a10d-b563941829e8 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:52.135757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fa9f61-b546-46bf-a10d-b563941829e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:52.136079 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fa9f61-b546-46bf-a10d-b563941829e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:52.137550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fa9f61-b546-46bf-a10d-b563941829e8 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:52.151810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 294/599] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:52.152712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fa9f61-b546-46bf-a10d-b563941829e8 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:52.248202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:52.263640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdad5f7-1b61-426f-ba0b-0b673361b6ef None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:52.272023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fa9f61-b546-46bf-a10d-b563941829e8 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:52.296025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:52.303267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924bfca3-ad41-455d-85dd-4bd8eca67b0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:52.304107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924bfca3-ad41-455d-85dd-4bd8eca67b0c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:52.304574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924bfca3-ad41-455d-85dd-4bd8eca67b0c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:52.305004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924bfca3-ad41-455d-85dd-4bd8eca67b0c None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:52.305993 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924bfca3-ad41-455d-85dd-4bd8eca67b0c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:52.306528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924bfca3-ad41-455d-85dd-4bd8eca67b0c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:52.308212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924bfca3-ad41-455d-85dd-4bd8eca67b0c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:52.308368 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2154d4a8-bca2-4680-9df1-4e49a821baf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:52.309099 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2154d4a8-bca2-4680-9df1-4e49a821baf8 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:52.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2154d4a8-bca2-4680-9df1-4e49a821baf8 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:52.309768 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2154d4a8-bca2-4680-9df1-4e49a821baf8 None admin] PATH_INFO: `/v3/projects/8b41573c9f3146d19a306089d686015c/users/5b0bd572ae7a4c7f9e07fe0b9708aa7e/roles/26925162cb82467f981eae19826e00c9` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:52.323223 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 306/600] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:52.324497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924bfca3-ad41-455d-85dd-4bd8eca67b0c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:52.326983 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2154d4a8-bca2-4680-9df1-4e49a821baf8 None admin] RBAC: Authorizing `identity:create_grant(project_id=8b41573c9f3146d19a306089d686015c, user_id=5b0bd572ae7a4c7f9e07fe0b9708aa7e, role_id=26925162cb82467f981eae19826e00c9)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:52.329413 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2154d4a8-bca2-4680-9df1-4e49a821baf8 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:52.351683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 295/601] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:54:52 2022] PUT /identity/v3/projects/8b41573c9f3146d19a306089d686015c/users/5b0bd572ae7a4c7f9e07fe0b9708aa7e/roles/26925162cb82467f981eae19826e00c9 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:52.352789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2154d4a8-bca2-4680-9df1-4e49a821baf8 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:52.443937 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924bfca3-ad41-455d-85dd-4bd8eca67b0c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:52.491549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9646a6c9-2088-4bc7-a428-42dc0bfaf318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:52.492546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9646a6c9-2088-4bc7-a428-42dc0bfaf318 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:52.493083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9646a6c9-2088-4bc7-a428-42dc0bfaf318 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:52.493529 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9646a6c9-2088-4bc7-a428-42dc0bfaf318 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:52.503005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9646a6c9-2088-4bc7-a428-42dc0bfaf318 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:52.504822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9646a6c9-2088-4bc7-a428-42dc0bfaf318 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-9646a6c9-2088-4bc7-a428-42dc0bfaf318 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:54:52.520080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:52.521052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:52.521916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:54:52.521916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:54:52.521916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:54:52.521916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:54:52.521916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:54:52.521916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:54:52.521916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:52.521916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:52.530379 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 307/602] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:52.537858 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9646a6c9-2088-4bc7-a428-42dc0bfaf318 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:52.563744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:52.578815 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2154d4a8-bca2-4680-9df1-4e49a821baf8 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:52.658809 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c430d74-44f8-4934-bd8b-53ef6bb4e26d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:52.659627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c430d74-44f8-4934-bd8b-53ef6bb4e26d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:52.659907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c430d74-44f8-4934-bd8b-53ef6bb4e26d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:52.660175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c430d74-44f8-4934-bd8b-53ef6bb4e26d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:52.668758 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c430d74-44f8-4934-bd8b-53ef6bb4e26d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:52.670391 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c430d74-44f8-4934-bd8b-53ef6bb4e26d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-8c430d74-44f8-4934-bd8b-53ef6bb4e26d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:54:52.686850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:52.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:52.688844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:54:52.688844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:54:52.688844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:54:52.688844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:54:52.688844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:54:52.688844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:54:52.688844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:52.688844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:52.689274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 296/603] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:52.695243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c430d74-44f8-4934-bd8b-53ef6bb4e26d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:52.717846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9646a6c9-2088-4bc7-a428-42dc0bfaf318 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:52.740275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:52.798841 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67574f90-ad51-4e24-97cf-3ff10b180cdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:52.801799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67574f90-ad51-4e24-97cf-3ff10b180cdd None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:52.802247 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67574f90-ad51-4e24-97cf-3ff10b180cdd None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:52.802684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67574f90-ad51-4e24-97cf-3ff10b180cdd None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:52.804401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67574f90-ad51-4e24-97cf-3ff10b180cdd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:52.804917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67574f90-ad51-4e24-97cf-3ff10b180cdd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:52.808878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67574f90-ad51-4e24-97cf-3ff10b180cdd None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:52.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 308/604] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:54:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:52.846939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67574f90-ad51-4e24-97cf-3ff10b180cdd None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:52.872006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c430d74-44f8-4934-bd8b-53ef6bb4e26d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:52.946262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9fe430-d515-48f3-8908-40eda087efaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:52.947190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9fe430-d515-48f3-8908-40eda087efaa None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:52.947190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9fe430-d515-48f3-8908-40eda087efaa None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:52.947477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9fe430-d515-48f3-8908-40eda087efaa None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:52.949967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9fe430-d515-48f3-8908-40eda087efaa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:52.949967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9fe430-d515-48f3-8908-40eda087efaa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:52.949967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9fe430-d515-48f3-8908-40eda087efaa None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:52.958632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:52.965501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 297/605] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:54:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:52.966947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c4e010-41e1-43c9-ae80-fb0233915d54 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:52.967233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c4e010-41e1-43c9-ae80-fb0233915d54 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:52.967553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c4e010-41e1-43c9-ae80-fb0233915d54 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:52.977991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 298/606] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:52.978465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c4e010-41e1-43c9-ae80-fb0233915d54 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:53.001627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67574f90-ad51-4e24-97cf-3ff10b180cdd None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:53.046580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8744482-00ed-4618-8ef0-602d87725cd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:53.047603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8744482-00ed-4618-8ef0-602d87725cd5 None admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:53.047885 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8744482-00ed-4618-8ef0-602d87725cd5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:53.048158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8744482-00ed-4618-8ef0-602d87725cd5 None admin] PATH_INFO: `/v3/projects/8b41573c9f3146d19a306089d686015c/users/9c8338bf95d34b228176b556afd93155/roles/26925162cb82467f981eae19826e00c9` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:53.073833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8744482-00ed-4618-8ef0-602d87725cd5 None admin] RBAC: Authorizing `identity:create_grant(project_id=8b41573c9f3146d19a306089d686015c, user_id=9c8338bf95d34b228176b556afd93155, role_id=26925162cb82467f981eae19826e00c9)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:53.075316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8744482-00ed-4618-8ef0-602d87725cd5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:53.094800 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:53.094997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 309/607] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:54:52 2022] PUT /identity/v3/projects/8b41573c9f3146d19a306089d686015c/users/9c8338bf95d34b228176b556afd93155/roles/26925162cb82467f981eae19826e00c9 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:53.096818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8d5a23-921e-4dd3-9a95-d778364b8a9f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:53.097110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8d5a23-921e-4dd3-9a95-d778364b8a9f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:53.097389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8d5a23-921e-4dd3-9a95-d778364b8a9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:53.106449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c4e010-41e1-43c9-ae80-fb0233915d54 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:53.122388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-2b8d5a23-921e-4dd3-9a95-d778364b8a9f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:53.124519 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2b8d5a23-921e-4dd3-9a95-d778364b8a9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:53.181738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0003b9-ca01-483c-9d0b-688036b13d68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:53.182536 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0003b9-ca01-483c-9d0b-688036b13d68 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:53.182824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0003b9-ca01-483c-9d0b-688036b13d68 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:53.183265 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0003b9-ca01-483c-9d0b-688036b13d68 None admin] PATH_INFO: `/v3/projects/8b41573c9f3146d19a306089d686015c/users/94e868131d33472b9ded8486274a07b9/roles/26925162cb82467f981eae19826e00c9` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:53.210983 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0003b9-ca01-483c-9d0b-688036b13d68 None admin] RBAC: Authorizing `identity:create_grant(project_id=8b41573c9f3146d19a306089d686015c, user_id=94e868131d33472b9ded8486274a07b9, role_id=26925162cb82467f981eae19826e00c9)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:53.212966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0003b9-ca01-483c-9d0b-688036b13d68 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:53.235449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 299/608] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:54:52 2022] PUT /identity/v3/projects/8b41573c9f3146d19a306089d686015c/users/94e868131d33472b9ded8486274a07b9/roles/26925162cb82467f981eae19826e00c9 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:53.237317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec1f1b3-d4e5-4aca-94e6-d294f9d8609c None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:53.237790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec1f1b3-d4e5-4aca-94e6-d294f9d8609c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:53.238410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec1f1b3-d4e5-4aca-94e6-d294f9d8609c None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:53.245785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 300/609] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:53 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:53.249923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 310/610] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:53.261933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e77dce-528e-4187-9a7d-e0bf8a9a1c0d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:53.262617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e77dce-528e-4187-9a7d-e0bf8a9a1c0d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:53.263276 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e77dce-528e-4187-9a7d-e0bf8a9a1c0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:53.274502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8d5a23-921e-4dd3-9a95-d778364b8a9f None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:53.300734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a7e77dce-528e-4187-9a7d-e0bf8a9a1c0d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:53.302696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a7e77dce-528e-4187-9a7d-e0bf8a9a1c0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:53.355631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:53.422260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 301/611] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:53.450967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8d5a23-921e-4dd3-9a95-d778364b8a9f None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:53.459745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e77dce-528e-4187-9a7d-e0bf8a9a1c0d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46555222-d109-46a7-a3bf-69315613ecb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:53.507057 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46555222-d109-46a7-a3bf-69315613ecb9 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:53.507334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46555222-d109-46a7-a3bf-69315613ecb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:53.507647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46555222-d109-46a7-a3bf-69315613ecb9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:53.527830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46555222-d109-46a7-a3bf-69315613ecb9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-46555222-d109-46a7-a3bf-69315613ecb9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:53.535655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:53.537235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:53.538774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:53.538774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:53.538774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:53.538774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:53.538774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:53.538774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:53.538774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:53.538774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:53.538774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:53.538774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:53.538774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:53.538774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:53.551496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 311/612] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:53.551496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46555222-d109-46a7-a3bf-69315613ecb9 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:53.606505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:53.673213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e77dce-528e-4187-9a7d-e0bf8a9a1c0d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:53.715685 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605ef1f-d6db-4e70-9ddc-7fd90544d196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:53.716715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605ef1f-d6db-4e70-9ddc-7fd90544d196 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:53.717245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605ef1f-d6db-4e70-9ddc-7fd90544d196 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:53.717740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605ef1f-d6db-4e70-9ddc-7fd90544d196 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:53.729378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605ef1f-d6db-4e70-9ddc-7fd90544d196 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-a605ef1f-d6db-4e70-9ddc-7fd90544d196 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:53.744839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:53.745836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:53.746633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:53.746633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:53.746633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:53.746633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:53.746633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:53.746633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:53.746633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:53.746633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:53.746633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:53.746633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:53.746633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:53.746633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:53.749439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 302/613] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:53.756872 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46555222-d109-46a7-a3bf-69315613ecb9 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:53.758571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605ef1f-d6db-4e70-9ddc-7fd90544d196 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:53.779799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:53.804180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0d587d-53e8-4827-bed4-ea1a1d5cfa9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:53.804866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0d587d-53e8-4827-bed4-ea1a1d5cfa9f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:53.805861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0d587d-53e8-4827-bed4-ea1a1d5cfa9f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:53.805861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0d587d-53e8-4827-bed4-ea1a1d5cfa9f None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:53.806132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0d587d-53e8-4827-bed4-ea1a1d5cfa9f None admin] RBAC: Adding query filter params (name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:53.806436 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0d587d-53e8-4827-bed4-ea1a1d5cfa9f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:53.807982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0d587d-53e8-4827-bed4-ea1a1d5cfa9f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:53.818940 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 312/614] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:53.831380 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0d587d-53e8-4827-bed4-ea1a1d5cfa9f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:53.929124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605ef1f-d6db-4e70-9ddc-7fd90544d196 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:53.957746 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0d587d-53e8-4827-bed4-ea1a1d5cfa9f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:53.968119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ce0765-cf04-440e-a3ae-5ff76b921f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:53.970467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ce0765-cf04-440e-a3ae-5ff76b921f12 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:53.971059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ce0765-cf04-440e-a3ae-5ff76b921f12 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:53.971822 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ce0765-cf04-440e-a3ae-5ff76b921f12 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:53.972761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ce0765-cf04-440e-a3ae-5ff76b921f12 None admin] RBAC: Adding query filter params (name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:53.973210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ce0765-cf04-440e-a3ae-5ff76b921f12 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:53.974772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ce0765-cf04-440e-a3ae-5ff76b921f12 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:53.976961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:53.987415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 303/615] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:53.993060 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ce0765-cf04-440e-a3ae-5ff76b921f12 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:54.000936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4922f534-406a-4e7a-aa2b-6ce869b32774 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:54.001582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4922f534-406a-4e7a-aa2b-6ce869b32774 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:54.001853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4922f534-406a-4e7a-aa2b-6ce869b32774 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:54.002120 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4922f534-406a-4e7a-aa2b-6ce869b32774 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:54.019574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4922f534-406a-4e7a-aa2b-6ce869b32774 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:54.023109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4922f534-406a-4e7a-aa2b-6ce869b32774 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-4922f534-406a-4e7a-aa2b-6ce869b32774 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:54.055538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:54.056659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:54.057675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:54.057675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:54.057675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:54.057675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:54.061255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 313/616] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:54.067853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4922f534-406a-4e7a-aa2b-6ce869b32774 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:54.114131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:54.194236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ce0765-cf04-440e-a3ae-5ff76b921f12 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:54.242539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fc6beb-c7f9-4049-a282-3771374c03a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:54.243482 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fc6beb-c7f9-4049-a282-3771374c03a3 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:54.244007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fc6beb-c7f9-4049-a282-3771374c03a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:54.244439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fc6beb-c7f9-4049-a282-3771374c03a3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:54.252589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4922f534-406a-4e7a-aa2b-6ce869b32774 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:54.269447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fc6beb-c7f9-4049-a282-3771374c03a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:54.271255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fc6beb-c7f9-4049-a282-3771374c03a3 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-04fc6beb-c7f9-4049-a282-3771374c03a3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:54.302297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:54.303434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:54.304502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:54.304502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:54.304502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:54.304502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:54.309663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 304/617] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:54:54.315323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fc6beb-c7f9-4049-a282-3771374c03a3 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:54.331788 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cac3c70-0438-4683-958e-0cd78207b520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:54.332654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cac3c70-0438-4683-958e-0cd78207b520 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:54.333102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cac3c70-0438-4683-958e-0cd78207b520 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:54.333515 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cac3c70-0438-4683-958e-0cd78207b520 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:54.334346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cac3c70-0438-4683-958e-0cd78207b520 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:54.334789 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cac3c70-0438-4683-958e-0cd78207b520 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:54.336363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cac3c70-0438-4683-958e-0cd78207b520 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:54.351878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 314/618] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:54.358298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cac3c70-0438-4683-958e-0cd78207b520 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:54.392012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:54.464893 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fc6beb-c7f9-4049-a282-3771374c03a3 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:54.512164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af44acab-afed-4477-8054-e320944f7fc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:54.514116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:54.515284 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af44acab-afed-4477-8054-e320944f7fc9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:54.515855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af44acab-afed-4477-8054-e320944f7fc9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:54.517361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af44acab-afed-4477-8054-e320944f7fc9 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:54.518456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af44acab-afed-4477-8054-e320944f7fc9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:54.519086 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af44acab-afed-4477-8054-e320944f7fc9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:54.522239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af44acab-afed-4477-8054-e320944f7fc9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:54.537663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 305/619] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:54.543410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af44acab-afed-4477-8054-e320944f7fc9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:54.560165 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cac3c70-0438-4683-958e-0cd78207b520 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:54.617549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfee679-1f61-4ff8-8340-74058cc36375 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:54.618722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfee679-1f61-4ff8-8340-74058cc36375 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:54.619169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfee679-1f61-4ff8-8340-74058cc36375 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:54.619494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfee679-1f61-4ff8-8340-74058cc36375 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:54.629341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfee679-1f61-4ff8-8340-74058cc36375 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:54.631478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfee679-1f61-4ff8-8340-74058cc36375 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-edfee679-1f61-4ff8-8340-74058cc36375 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:54.647251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:54.648280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:54.649222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:54.649222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:54.649222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:54.649222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:54.649222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:54.649222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:54.649222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:54.649222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:54.649222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 31 00:54:54.649222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:54.649222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 315/620] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:54 2022] GET /identity/v3/users/glance => generated 84 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:54.657860 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfee679-1f61-4ff8-8340-74058cc36375 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:54.769526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af44acab-afed-4477-8054-e320944f7fc9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:54.780935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:54.809327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b0a68-2a42-4c7f-ab54-84ae113e06b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:54.810157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b0a68-2a42-4c7f-ab54-84ae113e06b4 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:54.810547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b0a68-2a42-4c7f-ab54-84ae113e06b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:54.810913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b0a68-2a42-4c7f-ab54-84ae113e06b4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:54.819485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914b0a68-2a42-4c7f-ab54-84ae113e06b4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:54.821107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914b0a68-2a42-4c7f-ab54-84ae113e06b4 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:54.823278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfee679-1f61-4ff8-8340-74058cc36375 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-914b0a68-2a42-4c7f-ab54-84ae113e06b4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:54.835027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:54.836252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:54.837382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:54.837382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:54.837382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:54.837382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:54.837382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:54.837382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:54.837382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:54.837382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:54.837382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 31 00:54:54.837382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:54.838760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 306/621] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:54:54 2022] GET /identity/v3/users/swift => generated 83 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:54.848643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b0a68-2a42-4c7f-ab54-84ae113e06b4 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:54.874201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf76c7a9-e4f8-489d-920a-557b8ae55d17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:54.875103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf76c7a9-e4f8-489d-920a-557b8ae55d17 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:54.876936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf76c7a9-e4f8-489d-920a-557b8ae55d17 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:54.877519 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf76c7a9-e4f8-489d-920a-557b8ae55d17 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:54.880116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf76c7a9-e4f8-489d-920a-557b8ae55d17 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:54.881554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf76c7a9-e4f8-489d-920a-557b8ae55d17 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:54.883985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf76c7a9-e4f8-489d-920a-557b8ae55d17 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:54.911739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:54.920432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 316/622] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Jul 31 00:54:54 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:54.923220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf76c7a9-e4f8-489d-920a-557b8ae55d17 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:55.020765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b0a68-2a42-4c7f-ab54-84ae113e06b4 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:55.066073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d32c45e-bdea-4aa9-8a28-d5a6560d0b39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:55.066594 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d32c45e-bdea-4aa9-8a28-d5a6560d0b39 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:55.066867 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d32c45e-bdea-4aa9-8a28-d5a6560d0b39 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:55.067138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d32c45e-bdea-4aa9-8a28-d5a6560d0b39 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:55.067997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d32c45e-bdea-4aa9-8a28-d5a6560d0b39 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:55.068303 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d32c45e-bdea-4aa9-8a28-d5a6560d0b39 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:55.070428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d32c45e-bdea-4aa9-8a28-d5a6560d0b39 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:55.094252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 307/623] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Jul 31 00:54:54 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:55.096558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c82664e-a2b1-477c-871b-b35be1837175 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:55.096844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c82664e-a2b1-477c-871b-b35be1837175 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:55.097119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c82664e-a2b1-477c-871b-b35be1837175 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:55.099472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 308/624] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:55.101191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c82664e-a2b1-477c-871b-b35be1837175 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:55.105415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf76c7a9-e4f8-489d-920a-557b8ae55d17 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:55.137167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:55.152579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f65aa70-9cd5-4a5d-b9e2-425e1c405918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:55.153198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f65aa70-9cd5-4a5d-b9e2-425e1c405918 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:55.153488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f65aa70-9cd5-4a5d-b9e2-425e1c405918 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:55.153738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f65aa70-9cd5-4a5d-b9e2-425e1c405918 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:55.225321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c82664e-a2b1-477c-871b-b35be1837175 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:55.225543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:55.229050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f65aa70-9cd5-4a5d-b9e2-425e1c405918 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:55.230993 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f65aa70-9cd5-4a5d-b9e2-425e1c405918 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:55.268849 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e4e595-4d19-4054-a460-c17d48c632af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:55.269926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e4e595-4d19-4054-a460-c17d48c632af None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:55.270465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e4e595-4d19-4054-a460-c17d48c632af None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:55.271006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e4e595-4d19-4054-a460-c17d48c632af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:55.283191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e4e595-4d19-4054-a460-c17d48c632af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:55.285214 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e4e595-4d19-4054-a460-c17d48c632af None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-51e4e595-4d19-4054-a460-c17d48c632af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:55.306397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:55.307946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:55.309311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:55.309311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:55.309311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:55.309311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:55.312059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 309/625] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:55.314435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb6df19-4703-4d97-9030-aa1cd914fed8 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:55.315028 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb6df19-4703-4d97-9030-aa1cd914fed8 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:55.315536 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb6df19-4703-4d97-9030-aa1cd914fed8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:55.337522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-0cb6df19-4703-4d97-9030-aa1cd914fed8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:55.339087 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0cb6df19-4703-4d97-9030-aa1cd914fed8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:55.344675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 317/626] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:55.345690 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f65aa70-9cd5-4a5d-b9e2-425e1c405918 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:55.438141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 310/627] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:55.440099 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb6df19-4703-4d97-9030-aa1cd914fed8 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:55.452581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:55.468777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:55.474418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f65aa70-9cd5-4a5d-b9e2-425e1c405918 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:55.522476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69cbd65-e3f2-4210-8faf-485b77bf9c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:55.523389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69cbd65-e3f2-4210-8faf-485b77bf9c85 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:55.523812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69cbd65-e3f2-4210-8faf-485b77bf9c85 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:55.524217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69cbd65-e3f2-4210-8faf-485b77bf9c85 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:55.525097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69cbd65-e3f2-4210-8faf-485b77bf9c85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:55.525546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69cbd65-e3f2-4210-8faf-485b77bf9c85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:55.527099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69cbd65-e3f2-4210-8faf-485b77bf9c85 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:55.540931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 318/628] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:55.542131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69cbd65-e3f2-4210-8faf-485b77bf9c85 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:55.623909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb6df19-4703-4d97-9030-aa1cd914fed8 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:55.651339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69cbd65-e3f2-4210-8faf-485b77bf9c85 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:55.662635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94034d95-fc71-4d3d-8b17-0c489d72e943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:55.663457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94034d95-fc71-4d3d-8b17-0c489d72e943 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:55.663843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94034d95-fc71-4d3d-8b17-0c489d72e943 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:55.664218 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94034d95-fc71-4d3d-8b17-0c489d72e943 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:55.676857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94034d95-fc71-4d3d-8b17-0c489d72e943 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:55.678562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94034d95-fc71-4d3d-8b17-0c489d72e943 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:55.689916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abe4a13-3e3e-4df5-9d4a-69b1d59844db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:55.690785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abe4a13-3e3e-4df5-9d4a-69b1d59844db None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:55.691206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abe4a13-3e3e-4df5-9d4a-69b1d59844db None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:55.691638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abe4a13-3e3e-4df5-9d4a-69b1d59844db None admin] PATH_INFO: `/v3/roles/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-94034d95-fc71-4d3d-8b17-0c489d72e943 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:55.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:55.696700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:55.697719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:55.697719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:55.697719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:55.697719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:55.704000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 311/629] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:55.704000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94034d95-fc71-4d3d-8b17-0c489d72e943 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:55.711717 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abe4a13-3e3e-4df5-9d4a-69b1d59844db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-3abe4a13-3e3e-4df5-9d4a-69b1d59844db None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:55.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:55.718160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:55.719791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:55.719791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:55.719791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:55.719791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:55.719791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:55.719791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:55.719791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:55.719791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:55.719791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:55.719791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:55.719791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:55.719791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:55.723259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 319/630] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:55.724518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abe4a13-3e3e-4df5-9d4a-69b1d59844db None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:55.747399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:55.767719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:55.810560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94034d95-fc71-4d3d-8b17-0c489d72e943 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:55.835547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abe4a13-3e3e-4df5-9d4a-69b1d59844db None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:55.847144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1f61b4-77ab-41a7-a5b6-cdc81a583f0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:55.847973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1f61b4-77ab-41a7-a5b6-cdc81a583f0c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:55.848249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1f61b4-77ab-41a7-a5b6-cdc81a583f0c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:55.848521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1f61b4-77ab-41a7-a5b6-cdc81a583f0c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:55.855915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1f61b4-77ab-41a7-a5b6-cdc81a583f0c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:55.857405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1f61b4-77ab-41a7-a5b6-cdc81a583f0c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-bd1f61b4-77ab-41a7-a5b6-cdc81a583f0c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:54:55.869566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:55.870522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:55.871819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:54:55.871819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:54:55.871819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:54:55.871819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:54:55.871819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:54:55.871819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:54:55.871819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:55.871819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:55.871819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 312/631] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:55.872445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fdcd1a-d5fb-4da0-9e4e-bf3f157849e3 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:55.872445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fdcd1a-d5fb-4da0-9e4e-bf3f157849e3 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:55.872445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fdcd1a-d5fb-4da0-9e4e-bf3f157849e3 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:55.873672 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 313/632] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:55.874407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fdcd1a-d5fb-4da0-9e4e-bf3f157849e3 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:55.886976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f883b1-52e5-442c-9019-266fe8bc75d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:55.887957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f883b1-52e5-442c-9019-266fe8bc75d0 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:55.888245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f883b1-52e5-442c-9019-266fe8bc75d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:55.888529 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f883b1-52e5-442c-9019-266fe8bc75d0 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:55.889227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f883b1-52e5-442c-9019-266fe8bc75d0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:55.889531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f883b1-52e5-442c-9019-266fe8bc75d0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:55.890944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f883b1-52e5-442c-9019-266fe8bc75d0 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:55.904188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 320/633] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:55.905464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced7c6f-c27d-4b56-844b-148d222ae451 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:55.905745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced7c6f-c27d-4b56-844b-148d222ae451 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:55.906011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced7c6f-c27d-4b56-844b-148d222ae451 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:55.907887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 321/634] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:55.908579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ced7c6f-c27d-4b56-844b-148d222ae451 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:55.991749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fdcd1a-d5fb-4da0-9e4e-bf3f157849e3 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:55.995561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:56.012372 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:56.017733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ced7c6f-c27d-4b56-844b-148d222ae451 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:56.028021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd44bbe-d1a0-4ba8-9052-6bcc7389719a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:56.028654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd44bbe-d1a0-4ba8-9052-6bcc7389719a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:56.028909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd44bbe-d1a0-4ba8-9052-6bcc7389719a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:56.029176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd44bbe-d1a0-4ba8-9052-6bcc7389719a None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:56.029884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd44bbe-d1a0-4ba8-9052-6bcc7389719a None admin] RBAC: Adding query filter params (name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:56.030184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd44bbe-d1a0-4ba8-9052-6bcc7389719a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:56.031661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd44bbe-d1a0-4ba8-9052-6bcc7389719a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:56.047176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 314/635] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:56.047176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4308e5-9511-469a-a1c2-313c955a881c None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:56.047176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4308e5-9511-469a-a1c2-313c955a881c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:56.047176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4308e5-9511-469a-a1c2-313c955a881c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:56.055416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16673fb1-2b32-42bf-b5ed-515a37fa4c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:56.056037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16673fb1-2b32-42bf-b5ed-515a37fa4c40 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:56.056315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16673fb1-2b32-42bf-b5ed-515a37fa4c40 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:56.056583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16673fb1-2b32-42bf-b5ed-515a37fa4c40 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:56.057334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16673fb1-2b32-42bf-b5ed-515a37fa4c40 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:56.057639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16673fb1-2b32-42bf-b5ed-515a37fa4c40 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:56.059128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16673fb1-2b32-42bf-b5ed-515a37fa4c40 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:56.065526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ce4308e5-9511-469a-a1c2-313c955a881c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:56.066718 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ce4308e5-9511-469a-a1c2-313c955a881c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:56.071466 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 322/636] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:54:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:56.072293 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16673fb1-2b32-42bf-b5ed-515a37fa4c40 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:56.152125 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 315/637] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:56.154420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f8eadb-bfc3-46a5-8402-9a7d00804e04 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:56.155008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f8eadb-bfc3-46a5-8402-9a7d00804e04 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:56.155535 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f8eadb-bfc3-46a5-8402-9a7d00804e04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:56.185453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d1f8eadb-bfc3-46a5-8402-9a7d00804e04 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:56.187528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d1f8eadb-bfc3-46a5-8402-9a7d00804e04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:56.190295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16673fb1-2b32-42bf-b5ed-515a37fa4c40 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:56.228864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a77a69-0aa4-4a9b-990b-c75a381e3233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:56.229571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a77a69-0aa4-4a9b-990b-c75a381e3233 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:56.230034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a77a69-0aa4-4a9b-990b-c75a381e3233 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:56.230480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a77a69-0aa4-4a9b-990b-c75a381e3233 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:56.241744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a77a69-0aa4-4a9b-990b-c75a381e3233 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:56.244501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a77a69-0aa4-4a9b-990b-c75a381e3233 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:56.247110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-02a77a69-0aa4-4a9b-990b-c75a381e3233 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:54:56.263547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:56.264580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:56.265675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:54:56.265675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:54:56.265675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:54:56.265675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:54:56.265675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:54:56.265675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:54:56.265675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:56.265675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:56.265675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 323/638] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:56.265675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a77a69-0aa4-4a9b-990b-c75a381e3233 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:56.274129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 316/639] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:56.274951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f8eadb-bfc3-46a5-8402-9a7d00804e04 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:56.322772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:56.380710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a77a69-0aa4-4a9b-990b-c75a381e3233 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:56.382324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f8eadb-bfc3-46a5-8402-9a7d00804e04 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:56.416666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3959d006-b95c-4b7e-ab8f-3285b8b5c46f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:56.417227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3959d006-b95c-4b7e-ab8f-3285b8b5c46f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:56.417495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3959d006-b95c-4b7e-ab8f-3285b8b5c46f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:56.417763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3959d006-b95c-4b7e-ab8f-3285b8b5c46f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:56.418095 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8301e154-a02e-4d0c-bd72-72b76c108d20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:56.418797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8301e154-a02e-4d0c-bd72-72b76c108d20 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:56.419075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8301e154-a02e-4d0c-bd72-72b76c108d20 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:56.419340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8301e154-a02e-4d0c-bd72-72b76c108d20 None admin] PATH_INFO: `/v3/projects/8b41573c9f3146d19a306089d686015c/users/c11f1dd5a6b042c9b864a5861d0f3569/roles/26925162cb82467f981eae19826e00c9` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:56.429919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3959d006-b95c-4b7e-ab8f-3285b8b5c46f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:56.431418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3959d006-b95c-4b7e-ab8f-3285b8b5c46f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:56.441113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8301e154-a02e-4d0c-bd72-72b76c108d20 None admin] RBAC: Authorizing `identity:create_grant(project_id=8b41573c9f3146d19a306089d686015c, user_id=c11f1dd5a6b042c9b864a5861d0f3569, role_id=26925162cb82467f981eae19826e00c9)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:56.442623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8301e154-a02e-4d0c-bd72-72b76c108d20 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-3959d006-b95c-4b7e-ab8f-3285b8b5c46f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:56.446369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:56.448054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:56.448960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:56.448960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:56.448960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:56.448960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:56.448960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 324/640] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:56.448960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3959d006-b95c-4b7e-ab8f-3285b8b5c46f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:56.465558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 317/641] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:54:56 2022] PUT /identity/v3/projects/8b41573c9f3146d19a306089d686015c/users/c11f1dd5a6b042c9b864a5861d0f3569/roles/26925162cb82467f981eae19826e00c9 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:54:56.468306 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8301e154-a02e-4d0c-bd72-72b76c108d20 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:56.526867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:56.600516 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:56.601258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3959d006-b95c-4b7e-ab8f-3285b8b5c46f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:56.617297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8301e154-a02e-4d0c-bd72-72b76c108d20 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:56.651648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a80ecf-243e-47a3-9732-51d6cce5ecac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:56.652363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a80ecf-243e-47a3-9732-51d6cce5ecac None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:56.652645 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a80ecf-243e-47a3-9732-51d6cce5ecac None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:56.652913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a80ecf-243e-47a3-9732-51d6cce5ecac None admin] PATH_INFO: `/v3/roles/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:56.670854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a80ecf-243e-47a3-9732-51d6cce5ecac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:56.671083 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c21bf47-4609-435e-9b7a-7896aa6a9a6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:56.671414 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c21bf47-4609-435e-9b7a-7896aa6a9a6d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:56.671699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c21bf47-4609-435e-9b7a-7896aa6a9a6d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:56.671967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c21bf47-4609-435e-9b7a-7896aa6a9a6d None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:56.673225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c21bf47-4609-435e-9b7a-7896aa6a9a6d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:56.673524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c21bf47-4609-435e-9b7a-7896aa6a9a6d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:56.674946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c21bf47-4609-435e-9b7a-7896aa6a9a6d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-61a80ecf-243e-47a3-9732-51d6cce5ecac None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:56.682556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:56.683949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:56.685294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 325/642] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:56.691633 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a80ecf-243e-47a3-9732-51d6cce5ecac None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:56.691727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 318/643] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:54:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:56.692632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c21bf47-4609-435e-9b7a-7896aa6a9a6d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:56.825732 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a80ecf-243e-47a3-9732-51d6cce5ecac None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:56.831456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c21bf47-4609-435e-9b7a-7896aa6a9a6d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:56.846997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:56.871750 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a742b3-3198-4d78-b1b3-ab8b5a102280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:56.872387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a742b3-3198-4d78-b1b3-ab8b5a102280 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:56.872667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a742b3-3198-4d78-b1b3-ab8b5a102280 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:56.872941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a742b3-3198-4d78-b1b3-ab8b5a102280 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:56.883166 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a742b3-3198-4d78-b1b3-ab8b5a102280 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:56.884028 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f16b1b-1ef0-495c-87a0-7188dc3ac153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:56.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f16b1b-1ef0-495c-87a0-7188dc3ac153 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:56.885818 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f16b1b-1ef0-495c-87a0-7188dc3ac153 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:56.886468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f16b1b-1ef0-495c-87a0-7188dc3ac153 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:56.887866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f16b1b-1ef0-495c-87a0-7188dc3ac153 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-a4a742b3-3198-4d78-b1b3-ab8b5a102280 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:56.888485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:56.890228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 326/644] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:56.891867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a742b3-3198-4d78-b1b3-ab8b5a102280 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:56.893305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f16b1b-1ef0-495c-87a0-7188dc3ac153 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:56.893305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f16b1b-1ef0-495c-87a0-7188dc3ac153 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:56.923174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 319/645] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:56.924933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f16b1b-1ef0-495c-87a0-7188dc3ac153 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:56.951078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:57.004395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a742b3-3198-4d78-b1b3-ab8b5a102280 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:57.045964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8b1319-49c8-45be-9f65-1994a90fe175 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:57.046867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8b1319-49c8-45be-9f65-1994a90fe175 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:57.047894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8b1319-49c8-45be-9f65-1994a90fe175 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:57.048306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8b1319-49c8-45be-9f65-1994a90fe175 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:57.049478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8b1319-49c8-45be-9f65-1994a90fe175 None admin] RBAC: Adding query filter params (name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:57.049941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8b1319-49c8-45be-9f65-1994a90fe175 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:57.052490 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8b1319-49c8-45be-9f65-1994a90fe175 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:57.069967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 327/646] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:57.069967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8b1319-49c8-45be-9f65-1994a90fe175 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:57.077904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f16b1b-1ef0-495c-87a0-7188dc3ac153 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:57.114810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:57.120745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d2d0ae-18ea-490e-a75e-6432f0024418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:57.121364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d2d0ae-18ea-490e-a75e-6432f0024418 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:57.121639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d2d0ae-18ea-490e-a75e-6432f0024418 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:57.121909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d2d0ae-18ea-490e-a75e-6432f0024418 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:57.122692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d2d0ae-18ea-490e-a75e-6432f0024418 None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=4160039561434e51a87c6c7a019f2dff) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:57.122996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d2d0ae-18ea-490e-a75e-6432f0024418 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:57.125026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d2d0ae-18ea-490e-a75e-6432f0024418 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:57.135401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 320/647] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:56 2022] GET /identity/v3/role_assignments?user.id=4160039561434e51a87c6c7a019f2dff&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 608 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:57.135401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d2d0ae-18ea-490e-a75e-6432f0024418 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:57.226665 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:57.230130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8b1319-49c8-45be-9f65-1994a90fe175 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:57.253926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d2d0ae-18ea-490e-a75e-6432f0024418 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:57.278999 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab539033-ebff-467b-b46e-51323c21335b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:57.279764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab539033-ebff-467b-b46e-51323c21335b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:57.280146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab539033-ebff-467b-b46e-51323c21335b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:57.280482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab539033-ebff-467b-b46e-51323c21335b None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:57.281310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab539033-ebff-467b-b46e-51323c21335b None admin] RBAC: Adding query filter params (name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:57.282310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab539033-ebff-467b-b46e-51323c21335b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:57.284020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab539033-ebff-467b-b46e-51323c21335b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:57.298139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 328/648] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:57.298139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab539033-ebff-467b-b46e-51323c21335b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:57.308165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7438460-55cc-4b9d-87e0-3302d0213c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:57.309012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7438460-55cc-4b9d-87e0-3302d0213c98 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:57.309290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7438460-55cc-4b9d-87e0-3302d0213c98 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:57.310361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7438460-55cc-4b9d-87e0-3302d0213c98 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:57.320835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7438460-55cc-4b9d-87e0-3302d0213c98 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:57.322277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7438460-55cc-4b9d-87e0-3302d0213c98 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-c7438460-55cc-4b9d-87e0-3302d0213c98 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:57.334655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:57.335655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:57.337198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:57.337198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:57.337198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:57.337198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:57.337198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:57.337198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:57.337198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:57.337198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:57.337198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:54:57.337198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:57.337198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 321/649] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Jul 31 00:54:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:57.337198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7438460-55cc-4b9d-87e0-3302d0213c98 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:57.432400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:57.438518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab539033-ebff-467b-b46e-51323c21335b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:57.481204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11849a15-fe8f-404a-9a84-3fae07d0f42f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:57.481770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11849a15-fe8f-404a-9a84-3fae07d0f42f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:57.482050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11849a15-fe8f-404a-9a84-3fae07d0f42f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:57.482322 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11849a15-fe8f-404a-9a84-3fae07d0f42f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:57.483583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7438460-55cc-4b9d-87e0-3302d0213c98 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:57.497407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11849a15-fe8f-404a-9a84-3fae07d0f42f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:57.498950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11849a15-fe8f-404a-9a84-3fae07d0f42f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-11849a15-fe8f-404a-9a84-3fae07d0f42f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:57.519172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:57.520342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:57.521423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:57.521423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:57.521423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:57.521423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:57.523822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 329/650] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:57.524830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11849a15-fe8f-404a-9a84-3fae07d0f42f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:57.543786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2823ece-d182-46ca-89c5-f3800d06a488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:57.549607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2823ece-d182-46ca-89c5-f3800d06a488 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:57.550113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2823ece-d182-46ca-89c5-f3800d06a488 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:57.550613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2823ece-d182-46ca-89c5-f3800d06a488 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:57.554612 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:57.567299 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2823ece-d182-46ca-89c5-f3800d06a488 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:57.571654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2823ece-d182-46ca-89c5-f3800d06a488 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-b2823ece-d182-46ca-89c5-f3800d06a488 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:57.591012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:57.592190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:57.593093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:57.593093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:57.593093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:57.593093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:57.593093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 322/651] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:57.593374 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2823ece-d182-46ca-89c5-f3800d06a488 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:57.670838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11849a15-fe8f-404a-9a84-3fae07d0f42f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:57.717156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2823ece-d182-46ca-89c5-f3800d06a488 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:57.723804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b3b27e-29c3-48c4-83b2-e5a3046a4c07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:57.724446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b3b27e-29c3-48c4-83b2-e5a3046a4c07 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:57.724722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b3b27e-29c3-48c4-83b2-e5a3046a4c07 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:57.724993 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b3b27e-29c3-48c4-83b2-e5a3046a4c07 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:57.725802 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b3b27e-29c3-48c4-83b2-e5a3046a4c07 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:57.726105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b3b27e-29c3-48c4-83b2-e5a3046a4c07 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:57.728335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b3b27e-29c3-48c4-83b2-e5a3046a4c07 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:57.755029 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 330/652] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Jul 31 00:54:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:57.755836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b3b27e-29c3-48c4-83b2-e5a3046a4c07 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:57.766340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:57.768718 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e389c0e-8b28-480a-b351-a2ef1767f5af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:57.769352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e389c0e-8b28-480a-b351-a2ef1767f5af None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:57.769636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e389c0e-8b28-480a-b351-a2ef1767f5af None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:57.769911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e389c0e-8b28-480a-b351-a2ef1767f5af None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:57.770617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e389c0e-8b28-480a-b351-a2ef1767f5af None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:57.771393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e389c0e-8b28-480a-b351-a2ef1767f5af None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:57.773210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e389c0e-8b28-480a-b351-a2ef1767f5af None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:57.790627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 323/653] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:57.791882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e389c0e-8b28-480a-b351-a2ef1767f5af None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:57.860633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:57.884162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b3b27e-29c3-48c4-83b2-e5a3046a4c07 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:57.928160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bef7eaa-ee58-495e-a6c9-654428e8460b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:57.929421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bef7eaa-ee58-495e-a6c9-654428e8460b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:57.930055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bef7eaa-ee58-495e-a6c9-654428e8460b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:57.930510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bef7eaa-ee58-495e-a6c9-654428e8460b None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:57.931749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bef7eaa-ee58-495e-a6c9-654428e8460b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:57.932688 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bef7eaa-ee58-495e-a6c9-654428e8460b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:57.934293 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bef7eaa-ee58-495e-a6c9-654428e8460b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:57.940731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e389c0e-8b28-480a-b351-a2ef1767f5af None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:57.953030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 331/654] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:57.955072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bef7eaa-ee58-495e-a6c9-654428e8460b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:57.989753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27d8b7d-2774-4352-baeb-22814e9df4dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:57.990732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27d8b7d-2774-4352-baeb-22814e9df4dc None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:57.991520 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27d8b7d-2774-4352-baeb-22814e9df4dc None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:57.992424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27d8b7d-2774-4352-baeb-22814e9df4dc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:58.043231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:58.086423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bef7eaa-ee58-495e-a6c9-654428e8460b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:58.090899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27d8b7d-2774-4352-baeb-22814e9df4dc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:58.093883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27d8b7d-2774-4352-baeb-22814e9df4dc None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:58.125697 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b8a3e4-1dba-4720-ac76-cb8377605177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:58.127868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b8a3e4-1dba-4720-ac76-cb8377605177 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:58.128519 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b8a3e4-1dba-4720-ac76-cb8377605177 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:58.129614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b8a3e4-1dba-4720-ac76-cb8377605177 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:58.146705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b8a3e4-1dba-4720-ac76-cb8377605177 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:58.150031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b8a3e4-1dba-4720-ac76-cb8377605177 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:58.161001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-23b8a3e4-1dba-4720-ac76-cb8377605177 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:58.167855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:58.169527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:58.171255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:58.171255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:58.171255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:58.171255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:58.171255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:58.171255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:58.171255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:58.171255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:58.171255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 31 00:54:58.171255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:58.171255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 332/655] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:54:57 2022] GET /identity/v3/users/cinder => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:58.171255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b8a3e4-1dba-4720-ac76-cb8377605177 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:58.265253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 324/656] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 469 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:58.265253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27d8b7d-2774-4352-baeb-22814e9df4dc None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:58.347419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b8a3e4-1dba-4720-ac76-cb8377605177 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:58.390812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:58.398795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27d8b7d-2774-4352-baeb-22814e9df4dc None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:58.405009 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa0c91e-14d5-4b2c-8dde-faca98be20a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:58.405999 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0c91e-14d5-4b2c-8dde-faca98be20a3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:58.406518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0c91e-14d5-4b2c-8dde-faca98be20a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:58.407026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0c91e-14d5-4b2c-8dde-faca98be20a3 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:58.419038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa0c91e-14d5-4b2c-8dde-faca98be20a3 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:58.421515 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa0c91e-14d5-4b2c-8dde-faca98be20a3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-aaa0c91e-14d5-4b2c-8dde-faca98be20a3 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:54:58.438508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:58.439538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:58.440499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:58.440499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:58.440499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:54:58.440499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:54:58.440499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:54:58.440499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:54:58.440499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:54:58.440499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:54:58.440499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 31 00:54:58.440499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:58.441032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 333/657] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:58 2022] GET /identity/v3/users/neutron => generated 85 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:58.442087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa0c91e-14d5-4b2c-8dde-faca98be20a3 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:58.452884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc19e68-46a1-466e-b9d7-ce582de45591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:58.454495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc19e68-46a1-466e-b9d7-ce582de45591 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:58.454883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc19e68-46a1-466e-b9d7-ce582de45591 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:58.455308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc19e68-46a1-466e-b9d7-ce582de45591 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:58.456596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc19e68-46a1-466e-b9d7-ce582de45591 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:58.457113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc19e68-46a1-466e-b9d7-ce582de45591 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:58.460312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc19e68-46a1-466e-b9d7-ce582de45591 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:58.497134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 325/658] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Jul 31 00:54:58 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:58.499106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc19e68-46a1-466e-b9d7-ce582de45591 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:58.520272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:58.602494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa0c91e-14d5-4b2c-8dde-faca98be20a3 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:58.637887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d5430-c45d-40ac-a2b9-5eb0372939c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:58.638557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d5430-c45d-40ac-a2b9-5eb0372939c2 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:58.638850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d5430-c45d-40ac-a2b9-5eb0372939c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:58.639118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d5430-c45d-40ac-a2b9-5eb0372939c2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:58.650220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8d5430-c45d-40ac-a2b9-5eb0372939c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:58.651704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8d5430-c45d-40ac-a2b9-5eb0372939c2 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-ac8d5430-c45d-40ac-a2b9-5eb0372939c2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:58.667562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:58.668615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:58.669531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:58.669531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:58.669531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:58.669531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:58.669531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 334/659] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:58.669813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d5430-c45d-40ac-a2b9-5eb0372939c2 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:58.693267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc19e68-46a1-466e-b9d7-ce582de45591 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:58.725217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:58.732355 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54f926d-75e3-4531-bbe2-030dcdcb924f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:58.733121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54f926d-75e3-4531-bbe2-030dcdcb924f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:58.733499 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54f926d-75e3-4531-bbe2-030dcdcb924f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:58.733876 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54f926d-75e3-4531-bbe2-030dcdcb924f None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:58.734767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54f926d-75e3-4531-bbe2-030dcdcb924f None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:58.735180 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54f926d-75e3-4531-bbe2-030dcdcb924f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:58.736740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54f926d-75e3-4531-bbe2-030dcdcb924f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:58.762955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 326/660] 10.222.0.42 () {56 vars in 1139 bytes} [Sun Jul 31 00:54:58 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:58.762955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6b5f38-dc3c-45f2-b1ba-6437bd3c0c0a None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:58.762955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6b5f38-dc3c-45f2-b1ba-6437bd3c0c0a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:58.762955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6b5f38-dc3c-45f2-b1ba-6437bd3c0c0a None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:58.762955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 327/661] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:58.762955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6b5f38-dc3c-45f2-b1ba-6437bd3c0c0a None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:58.808488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d5430-c45d-40ac-a2b9-5eb0372939c2 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:58.834884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:58.852034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bbe620-494a-4a22-bda3-eab0ec57707e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:58.852723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bbe620-494a-4a22-bda3-eab0ec57707e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:58.852998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bbe620-494a-4a22-bda3-eab0ec57707e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:58.853265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bbe620-494a-4a22-bda3-eab0ec57707e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:58.893222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6b5f38-dc3c-45f2-b1ba-6437bd3c0c0a None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:58.933683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e966d180-0a8b-4f76-a7dd-f93c8fce1e2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:58.934307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e966d180-0a8b-4f76-a7dd-f93c8fce1e2b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:58.934585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e966d180-0a8b-4f76-a7dd-f93c8fce1e2b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:58.934862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e966d180-0a8b-4f76-a7dd-f93c8fce1e2b None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:58.935592 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e966d180-0a8b-4f76-a7dd-f93c8fce1e2b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:58.935894 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e966d180-0a8b-4f76-a7dd-f93c8fce1e2b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:58.937332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e966d180-0a8b-4f76-a7dd-f93c8fce1e2b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:58.949961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bbe620-494a-4a22-bda3-eab0ec57707e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:58.950693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 328/662] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:58.951580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e966d180-0a8b-4f76-a7dd-f93c8fce1e2b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:58.954244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bbe620-494a-4a22-bda3-eab0ec57707e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:59.014813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:59.070687 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e966d180-0a8b-4f76-a7dd-f93c8fce1e2b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:59.084483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 335/663] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:59.084483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf78d73-c048-45d9-bcd9-df6112e1bbea None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:59.084483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf78d73-c048-45d9-bcd9-df6112e1bbea None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:59.084483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf78d73-c048-45d9-bcd9-df6112e1bbea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:59.099738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-dbf78d73-c048-45d9-bcd9-df6112e1bbea None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:59.101013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-dbf78d73-c048-45d9-bcd9-df6112e1bbea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:59.101147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:59.116547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee18d609-7708-4422-aabd-7a11f8bb7a66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:59.117395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee18d609-7708-4422-aabd-7a11f8bb7a66 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:59.117847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee18d609-7708-4422-aabd-7a11f8bb7a66 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:59.118394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee18d609-7708-4422-aabd-7a11f8bb7a66 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:59.173029 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 336/664] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:59.173872 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf78d73-c048-45d9-bcd9-df6112e1bbea None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:59.199787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee18d609-7708-4422-aabd-7a11f8bb7a66 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:59.201249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee18d609-7708-4422-aabd-7a11f8bb7a66 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:59.263186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:59.314534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf78d73-c048-45d9-bcd9-df6112e1bbea None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:59.330162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 329/665] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:54:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:54:59.335542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee18d609-7708-4422-aabd-7a11f8bb7a66 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:59.363938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:59.364522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a35c6c4-ef6f-4499-9e9d-fcf28ab86ad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:59.365443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a35c6c4-ef6f-4499-9e9d-fcf28ab86ad5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:59.365719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a35c6c4-ef6f-4499-9e9d-fcf28ab86ad5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:59.366437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a35c6c4-ef6f-4499-9e9d-fcf28ab86ad5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:59.374918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a35c6c4-ef6f-4499-9e9d-fcf28ab86ad5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:59.377063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a35c6c4-ef6f-4499-9e9d-fcf28ab86ad5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-8a35c6c4-ef6f-4499-9e9d-fcf28ab86ad5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:54:59.392390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:59.393525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:59.394550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:54:59.394550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:54:59.394550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:54:59.394550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:54:59.394550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:54:59.394550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:54:59.394550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:54:59.394550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:59.394550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 337/666] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:54:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:59.395078 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a35c6c4-ef6f-4499-9e9d-fcf28ab86ad5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:59.455464 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee18d609-7708-4422-aabd-7a11f8bb7a66 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:59.489698 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e9d156-87ab-41f1-9496-2aef11205eb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:59.490332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e9d156-87ab-41f1-9496-2aef11205eb1 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:59.490607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e9d156-87ab-41f1-9496-2aef11205eb1 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:59.490886 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e9d156-87ab-41f1-9496-2aef11205eb1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:59.495601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a35c6c4-ef6f-4499-9e9d-fcf28ab86ad5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:59.501242 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e9d156-87ab-41f1-9496-2aef11205eb1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:59.502876 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e9d156-87ab-41f1-9496-2aef11205eb1 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-03e9d156-87ab-41f1-9496-2aef11205eb1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:59.518048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:59.518991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:59.520334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:59.520334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:59.520334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:59.520334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:59.520334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 330/667] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:59.520334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e9d156-87ab-41f1-9496-2aef11205eb1 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:59.528348 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:59.536459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611c2894-42ed-4b49-92fa-ca215468a7d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:59.537095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611c2894-42ed-4b49-92fa-ca215468a7d6 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:59.537366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611c2894-42ed-4b49-92fa-ca215468a7d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:59.537631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611c2894-42ed-4b49-92fa-ca215468a7d6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:59.550747 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611c2894-42ed-4b49-92fa-ca215468a7d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-611c2894-42ed-4b49-92fa-ca215468a7d6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:54:59.555878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:59.556814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:54:59.557649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 338/668] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:54:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:59.558319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835083d2-236f-4abc-a180-a86567698d9c None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:59.558446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835083d2-236f-4abc-a180-a86567698d9c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:59.558844 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835083d2-236f-4abc-a180-a86567698d9c None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:59.559975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 339/669] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:54:59 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:54:59.560642 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835083d2-236f-4abc-a180-a86567698d9c None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:59.609145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:59.634219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e9d156-87ab-41f1-9496-2aef11205eb1 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:59.667317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e694697d-b617-4868-8253-6c519f0b50cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:59.667587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835083d2-236f-4abc-a180-a86567698d9c None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:59.668125 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e694697d-b617-4868-8253-6c519f0b50cf None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:59.668466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e694697d-b617-4868-8253-6c519f0b50cf None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:59.668809 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e694697d-b617-4868-8253-6c519f0b50cf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:59.680199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e694697d-b617-4868-8253-6c519f0b50cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:59.681667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e694697d-b617-4868-8253-6c519f0b50cf None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-e694697d-b617-4868-8253-6c519f0b50cf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:54:59.697233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:54:59.698307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:54:59.699268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:54:59.699268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:54:59.699268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:54:59.699268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:54:59.699268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 331/670] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:54:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:54:59.699268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e694697d-b617-4868-8253-6c519f0b50cf None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:59.704659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb62b0f-55ff-4aaa-b352-beabaf7dbcde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:59.705271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb62b0f-55ff-4aaa-b352-beabaf7dbcde None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:59.705547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb62b0f-55ff-4aaa-b352-beabaf7dbcde None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:59.705816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb62b0f-55ff-4aaa-b352-beabaf7dbcde None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:59.706563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb62b0f-55ff-4aaa-b352-beabaf7dbcde None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:59.706866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb62b0f-55ff-4aaa-b352-beabaf7dbcde None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:59.708326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb62b0f-55ff-4aaa-b352-beabaf7dbcde None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:59.720082 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 340/671] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:54:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:59.720892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb62b0f-55ff-4aaa-b352-beabaf7dbcde None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:59.770710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:59.799057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e694697d-b617-4868-8253-6c519f0b50cf None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:59.821856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb62b0f-55ff-4aaa-b352-beabaf7dbcde None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:59.825005 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:54:59.833052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e94bfb3-13ab-482c-880b-138ccade289c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:59.833659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e94bfb3-13ab-482c-880b-138ccade289c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:59.833930 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e94bfb3-13ab-482c-880b-138ccade289c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:59.834197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e94bfb3-13ab-482c-880b-138ccade289c None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:59.834877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e94bfb3-13ab-482c-880b-138ccade289c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:59.835174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e94bfb3-13ab-482c-880b-138ccade289c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:59.836555 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e94bfb3-13ab-482c-880b-138ccade289c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:59.849318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 332/672] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:59.850501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768f90f-c756-4e55-8515-a76d3cb6e3e8 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:59.850780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768f90f-c756-4e55-8515-a76d3cb6e3e8 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:59.851051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768f90f-c756-4e55-8515-a76d3cb6e3e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:59.856769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c399ad8e-2753-4ac2-8dc2-76bb794b375c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:54:59.857385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c399ad8e-2753-4ac2-8dc2-76bb794b375c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:54:59.857655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c399ad8e-2753-4ac2-8dc2-76bb794b375c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:54:59.857922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c399ad8e-2753-4ac2-8dc2-76bb794b375c None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:54:59.858633 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c399ad8e-2753-4ac2-8dc2-76bb794b375c None admin] RBAC: Adding query filter params (name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:54:59.858927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c399ad8e-2753-4ac2-8dc2-76bb794b375c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:54:59.860956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c399ad8e-2753-4ac2-8dc2-76bb794b375c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:54:59.869399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 341/673] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:54:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:54:59.869943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-1768f90f-c756-4e55-8515-a76d3cb6e3e8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:54:59.870191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c399ad8e-2753-4ac2-8dc2-76bb794b375c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:59.871143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1768f90f-c756-4e55-8515-a76d3cb6e3e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:54:59.937979 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 333/674] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:54:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:54:59.938820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1768f90f-c756-4e55-8515-a76d3cb6e3e8 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:54:59.970077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c399ad8e-2753-4ac2-8dc2-76bb794b375c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:54:59.980576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:00.005242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f37aef-1e26-4c45-88d2-21bb419cc317 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:00.005861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f37aef-1e26-4c45-88d2-21bb419cc317 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:00.006145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f37aef-1e26-4c45-88d2-21bb419cc317 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:00.006411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f37aef-1e26-4c45-88d2-21bb419cc317 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:00.007103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f37aef-1e26-4c45-88d2-21bb419cc317 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:00.007435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f37aef-1e26-4c45-88d2-21bb419cc317 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:00.008797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f37aef-1e26-4c45-88d2-21bb419cc317 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:00.020531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 342/675] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:54:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:00.021344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f37aef-1e26-4c45-88d2-21bb419cc317 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:00.040622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1768f90f-c756-4e55-8515-a76d3cb6e3e8 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:00.059169 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:00.076150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220c408d-8109-48b3-a5bb-dc4cd7370be4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:00.076763 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220c408d-8109-48b3-a5bb-dc4cd7370be4 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:00.077037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220c408d-8109-48b3-a5bb-dc4cd7370be4 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:00.077360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220c408d-8109-48b3-a5bb-dc4cd7370be4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:00.078139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220c408d-8109-48b3-a5bb-dc4cd7370be4 None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=5b0bd572ae7a4c7f9e07fe0b9708aa7e) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:00.078512 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220c408d-8109-48b3-a5bb-dc4cd7370be4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:00.079964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220c408d-8109-48b3-a5bb-dc4cd7370be4 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:00.085765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 334/676] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:54:59 2022] GET /identity/v3/role_assignments?user.id=5b0bd572ae7a4c7f9e07fe0b9708aa7e&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:00.085765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220c408d-8109-48b3-a5bb-dc4cd7370be4 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:00.133260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f37aef-1e26-4c45-88d2-21bb419cc317 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:00.175406 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50874778-6028-45f7-bb59-cf0c535be33f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:00.176350 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50874778-6028-45f7-bb59-cf0c535be33f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:00.176678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50874778-6028-45f7-bb59-cf0c535be33f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:00.176878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50874778-6028-45f7-bb59-cf0c535be33f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:00.185301 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50874778-6028-45f7-bb59-cf0c535be33f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:00.186800 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50874778-6028-45f7-bb59-cf0c535be33f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-50874778-6028-45f7-bb59-cf0c535be33f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:00.199495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:00.200429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:00.207792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:00.207792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:00.207792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:00.207792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:00.207792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:00.207792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:00.207792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:00.207792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:00.207792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 343/677] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:00.207792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50874778-6028-45f7-bb59-cf0c535be33f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:00.221330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220c408d-8109-48b3-a5bb-dc4cd7370be4 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:00.239697 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:00.267034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c95714-b45c-45aa-931f-9dfd1b78e022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:00.267855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c95714-b45c-45aa-931f-9dfd1b78e022 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:00.268918 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c95714-b45c-45aa-931f-9dfd1b78e022 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:00.268918 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c95714-b45c-45aa-931f-9dfd1b78e022 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:00.280171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c95714-b45c-45aa-931f-9dfd1b78e022 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:00.282463 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c95714-b45c-45aa-931f-9dfd1b78e022 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-83c95714-b45c-45aa-931f-9dfd1b78e022 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:00.298562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:00.299549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:00.300467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:00.300467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:00.300467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:00.300467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:00.300467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 335/678] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:00.300744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c95714-b45c-45aa-931f-9dfd1b78e022 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:00.336665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50874778-6028-45f7-bb59-cf0c535be33f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:00.342167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:00.374675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2949bcb-efda-4667-96fd-21005dd85b49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:00.376161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2949bcb-efda-4667-96fd-21005dd85b49 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:00.376656 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2949bcb-efda-4667-96fd-21005dd85b49 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:00.377113 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2949bcb-efda-4667-96fd-21005dd85b49 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:00.389612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2949bcb-efda-4667-96fd-21005dd85b49 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:00.391645 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2949bcb-efda-4667-96fd-21005dd85b49 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-d2949bcb-efda-4667-96fd-21005dd85b49 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:00.407405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:00.408368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:00.409236 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:00.409236 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:00.409236 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:00.409236 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:00.409236 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 344/679] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:00.409509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2949bcb-efda-4667-96fd-21005dd85b49 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:00.428782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c95714-b45c-45aa-931f-9dfd1b78e022 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:00.466787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7899eab3-55ec-4ac8-a396-ce6d03bb1021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:00.467371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7899eab3-55ec-4ac8-a396-ce6d03bb1021 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:00.467654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7899eab3-55ec-4ac8-a396-ce6d03bb1021 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:00.467931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7899eab3-55ec-4ac8-a396-ce6d03bb1021 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:00.475884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7899eab3-55ec-4ac8-a396-ce6d03bb1021 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:00.477340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7899eab3-55ec-4ac8-a396-ce6d03bb1021 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-7899eab3-55ec-4ac8-a396-ce6d03bb1021 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:00.490158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:00.491294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:00.492600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:00.492600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:00.492600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:00.492600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:00.492600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:00.492600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:00.492600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:00.492600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:00.492600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 336/680] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:00.492600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7899eab3-55ec-4ac8-a396-ce6d03bb1021 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:00.511480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:00.532726 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2949bcb-efda-4667-96fd-21005dd85b49 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:00.576016 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32f8a00-4a7c-4262-b9a8-a773333c8834 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:00.576640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32f8a00-4a7c-4262-b9a8-a773333c8834 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:00.576917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32f8a00-4a7c-4262-b9a8-a773333c8834 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:00.577189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32f8a00-4a7c-4262-b9a8-a773333c8834 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:00.577992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32f8a00-4a7c-4262-b9a8-a773333c8834 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:00.578296 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32f8a00-4a7c-4262-b9a8-a773333c8834 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:00.580299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32f8a00-4a7c-4262-b9a8-a773333c8834 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:00.593250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 345/681] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:00.594644 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32f8a00-4a7c-4262-b9a8-a773333c8834 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:00.609132 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:00.618214 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7899eab3-55ec-4ac8-a396-ce6d03bb1021 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:00.661680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad4aa04-2222-4130-ba96-36ec91e350cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:00.662349 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad4aa04-2222-4130-ba96-36ec91e350cd None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:00.662614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad4aa04-2222-4130-ba96-36ec91e350cd None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:00.663034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad4aa04-2222-4130-ba96-36ec91e350cd None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:00.663750 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad4aa04-2222-4130-ba96-36ec91e350cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:00.664054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad4aa04-2222-4130-ba96-36ec91e350cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:00.665591 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad4aa04-2222-4130-ba96-36ec91e350cd None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:00.679749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 337/682] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:00.680640 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad4aa04-2222-4130-ba96-36ec91e350cd None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:00.716157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32f8a00-4a7c-4262-b9a8-a773333c8834 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:00.753665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2447b95d-bfb0-43c3-9808-b5d008586e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:00.754201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2447b95d-bfb0-43c3-9808-b5d008586e3e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:00.754482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2447b95d-bfb0-43c3-9808-b5d008586e3e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:00.754853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2447b95d-bfb0-43c3-9808-b5d008586e3e None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:00.755476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2447b95d-bfb0-43c3-9808-b5d008586e3e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:00.755783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2447b95d-bfb0-43c3-9808-b5d008586e3e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:00.757231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2447b95d-bfb0-43c3-9808-b5d008586e3e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:00.773522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 346/683] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:00.773522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2447b95d-bfb0-43c3-9808-b5d008586e3e None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:00.773522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:00.798548 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad4aa04-2222-4130-ba96-36ec91e350cd None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:00.836334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2588721d-54f7-4ad5-a919-3c9ad83eb57b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:00.838285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2588721d-54f7-4ad5-a919-3c9ad83eb57b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:00.838822 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2588721d-54f7-4ad5-a919-3c9ad83eb57b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:00.839223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2588721d-54f7-4ad5-a919-3c9ad83eb57b None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:00.840167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2588721d-54f7-4ad5-a919-3c9ad83eb57b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:00.840637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2588721d-54f7-4ad5-a919-3c9ad83eb57b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:00.843015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2588721d-54f7-4ad5-a919-3c9ad83eb57b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:00.862895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 338/684] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:00.862895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2588721d-54f7-4ad5-a919-3c9ad83eb57b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:00.877841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:00.885363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2447b95d-bfb0-43c3-9808-b5d008586e3e None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:00.921196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c31268-a192-4ded-9f38-53510b141558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:00.921826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c31268-a192-4ded-9f38-53510b141558 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:00.922133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c31268-a192-4ded-9f38-53510b141558 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:00.922411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c31268-a192-4ded-9f38-53510b141558 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:00.923198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c31268-a192-4ded-9f38-53510b141558 None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=9c8338bf95d34b228176b556afd93155) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:00.923541 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c31268-a192-4ded-9f38-53510b141558 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:00.924975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c31268-a192-4ded-9f38-53510b141558 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:00.939393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 347/685] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:00 2022] GET /identity/v3/role_assignments?user.id=9c8338bf95d34b228176b556afd93155&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:00.939393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c31268-a192-4ded-9f38-53510b141558 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:00.984100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2588721d-54f7-4ad5-a919-3c9ad83eb57b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:01.025889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e84003-a764-4330-8b65-b9366614c4ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:01.026646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e84003-a764-4330-8b65-b9366614c4ec None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:01.026999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e84003-a764-4330-8b65-b9366614c4ec None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:01.027341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e84003-a764-4330-8b65-b9366614c4ec None admin] PATH_INFO: `/v3/users/swift` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:01.036792 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e84003-a764-4330-8b65-b9366614c4ec None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:01.038481 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e84003-a764-4330-8b65-b9366614c4ec None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-a4e84003-a764-4330-8b65-b9366614c4ec None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:01.053341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:01.054543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:01.055728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:01.055728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:01.055728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:01.055728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:01.055728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:01.055728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:01.055728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:01.055728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:01.055728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 31 00:55:01.055728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:01.057958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 339/686] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:00 2022] GET /identity/v3/users/swift => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:01.057958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e84003-a764-4330-8b65-b9366614c4ec None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:01.148532 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:01.160114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:01.196552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c31268-a192-4ded-9f38-53510b141558 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:01.203602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e84003-a764-4330-8b65-b9366614c4ec None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:01.247275 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb5391d-3f10-422b-a411-e35bb7f1b7cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:01.247934 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb5391d-3f10-422b-a411-e35bb7f1b7cc None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:01.248216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb5391d-3f10-422b-a411-e35bb7f1b7cc None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:01.248594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb5391d-3f10-422b-a411-e35bb7f1b7cc None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:01.249393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb5391d-3f10-422b-a411-e35bb7f1b7cc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:01.251467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb5391d-3f10-422b-a411-e35bb7f1b7cc None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:01.254726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bc50fe-91c5-476c-89aa-a6b9f0f0dc27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:01.255399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bc50fe-91c5-476c-89aa-a6b9f0f0dc27 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:01.255650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bc50fe-91c5-476c-89aa-a6b9f0f0dc27 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:01.255917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bc50fe-91c5-476c-89aa-a6b9f0f0dc27 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:01.256702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bc50fe-91c5-476c-89aa-a6b9f0f0dc27 None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=94e868131d33472b9ded8486274a07b9) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:01.257006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bc50fe-91c5-476c-89aa-a6b9f0f0dc27 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:01.258425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bc50fe-91c5-476c-89aa-a6b9f0f0dc27 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:01.267000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 340/687] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:01 2022] GET /identity/v3/role_assignments?user.id=94e868131d33472b9ded8486274a07b9&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 608 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:01.267963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bc50fe-91c5-476c-89aa-a6b9f0f0dc27 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:01.287822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 348/688] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:55:00 2022] POST /identity/v3/users => generated 254 bytes in 352 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:01.436249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bc50fe-91c5-476c-89aa-a6b9f0f0dc27 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:01.494107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e14825-e291-42a6-86a8-1da061b6fc65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:01.494734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e14825-e291-42a6-86a8-1da061b6fc65 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:01.495017 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e14825-e291-42a6-86a8-1da061b6fc65 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:01.495291 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e14825-e291-42a6-86a8-1da061b6fc65 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:01.496189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e14825-e291-42a6-86a8-1da061b6fc65 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:01.496500 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e14825-e291-42a6-86a8-1da061b6fc65 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:01.497939 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e14825-e291-42a6-86a8-1da061b6fc65 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:01.508859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:01.530753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 341/689] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Jul 31 00:55:01 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:01.532986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb5391d-3f10-422b-a411-e35bb7f1b7cc None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:01.655829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb5391d-3f10-422b-a411-e35bb7f1b7cc None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:01.680607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:01.694954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af32d02-4bb1-4fcf-a0b6-d81f12eb3d36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:01.695617 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af32d02-4bb1-4fcf-a0b6-d81f12eb3d36 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:01.695896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af32d02-4bb1-4fcf-a0b6-d81f12eb3d36 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:01.696169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af32d02-4bb1-4fcf-a0b6-d81f12eb3d36 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:01.770985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af32d02-4bb1-4fcf-a0b6-d81f12eb3d36 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:01.772479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af32d02-4bb1-4fcf-a0b6-d81f12eb3d36 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:01.890446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 349/690] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:55:01.896613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e14825-e291-42a6-86a8-1da061b6fc65 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:02.021620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e14825-e291-42a6-86a8-1da061b6fc65 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:02.070011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b9de77-113c-4243-8c11-9335fd92480e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:02.070674 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b9de77-113c-4243-8c11-9335fd92480e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:02.070953 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b9de77-113c-4243-8c11-9335fd92480e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:02.071227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b9de77-113c-4243-8c11-9335fd92480e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:02.087518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b9de77-113c-4243-8c11-9335fd92480e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:02.088995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b9de77-113c-4243-8c11-9335fd92480e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-03b9de77-113c-4243-8c11-9335fd92480e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:02.104950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:02.105864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:02.106735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:02.106735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:02.106735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:02.106735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:02.106735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 342/691] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:02.111811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af32d02-4bb1-4fcf-a0b6-d81f12eb3d36 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:02.152897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:02.274305 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af32d02-4bb1-4fcf-a0b6-d81f12eb3d36 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:02.297490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feee98d6-8e14-4f47-a568-e66c9f0304f3 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:02.298009 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feee98d6-8e14-4f47-a568-e66c9f0304f3 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:02.298009 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feee98d6-8e14-4f47-a568-e66c9f0304f3 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:02.307379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 343/692] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:02.307379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89c37ad-7b2f-4260-acc2-b95f3bad9e23 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:02.307379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89c37ad-7b2f-4260-acc2-b95f3bad9e23 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:02.307379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89c37ad-7b2f-4260-acc2-b95f3bad9e23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:02.321451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcf9e1e-8787-4b2d-832b-df31956f27cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:02.322073 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcf9e1e-8787-4b2d-832b-df31956f27cd None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:02.322353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcf9e1e-8787-4b2d-832b-df31956f27cd None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:02.322626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcf9e1e-8787-4b2d-832b-df31956f27cd None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:02.323409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcf9e1e-8787-4b2d-832b-df31956f27cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:02.323721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcf9e1e-8787-4b2d-832b-df31956f27cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:02.325208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcf9e1e-8787-4b2d-832b-df31956f27cd None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:02.338381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 350/693] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:02.344334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcf9e1e-8787-4b2d-832b-df31956f27cd None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:02.346433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-e89c37ad-7b2f-4260-acc2-b95f3bad9e23 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:02.347642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e89c37ad-7b2f-4260-acc2-b95f3bad9e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:02.417909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:02.454574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 344/694] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:55:02.476855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89c37ad-7b2f-4260-acc2-b95f3bad9e23 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:02.517399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:02.558124 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcf9e1e-8787-4b2d-832b-df31956f27cd None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:02.640210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf3289e-18b6-40ba-95cb-f07953267d59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:02.640912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf3289e-18b6-40ba-95cb-f07953267d59 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:02.641195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf3289e-18b6-40ba-95cb-f07953267d59 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:02.641467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf3289e-18b6-40ba-95cb-f07953267d59 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:02.657717 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf3289e-18b6-40ba-95cb-f07953267d59 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:02.659195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf3289e-18b6-40ba-95cb-f07953267d59 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:02.681925 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89c37ad-7b2f-4260-acc2-b95f3bad9e23 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-faf3289e-18b6-40ba-95cb-f07953267d59 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:02.685157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:02.686031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:02.686960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:02.686960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:02.686960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:02.686960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:02.686960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:02.686960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:02.686960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:02.686960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:02.686960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 351/695] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 343 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:02.693073 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf3289e-18b6-40ba-95cb-f07953267d59 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:02.751061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90593bf7-60b7-4cf4-9aba-6a62fd9549d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:02.751736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90593bf7-60b7-4cf4-9aba-6a62fd9549d6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:02.752010 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90593bf7-60b7-4cf4-9aba-6a62fd9549d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:02.752279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90593bf7-60b7-4cf4-9aba-6a62fd9549d6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:02.778688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90593bf7-60b7-4cf4-9aba-6a62fd9549d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-90593bf7-60b7-4cf4-9aba-6a62fd9549d6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:02.790371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:02.792926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:02.794885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 345/696] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:02.803111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90593bf7-60b7-4cf4-9aba-6a62fd9549d6 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:02.879304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:02.895725 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf3289e-18b6-40ba-95cb-f07953267d59 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:02.968125 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90593bf7-60b7-4cf4-9aba-6a62fd9549d6 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:02.976895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08459a5-c215-45f5-9e8c-52b45d60c4f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:02.977528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08459a5-c215-45f5-9e8c-52b45d60c4f3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:02.977801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08459a5-c215-45f5-9e8c-52b45d60c4f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:02.978069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08459a5-c215-45f5-9e8c-52b45d60c4f3 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:02.978839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08459a5-c215-45f5-9e8c-52b45d60c4f3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:02.979146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08459a5-c215-45f5-9e8c-52b45d60c4f3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:02.980704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08459a5-c215-45f5-9e8c-52b45d60c4f3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:02.995091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 352/697] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:03.004557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08459a5-c215-45f5-9e8c-52b45d60c4f3 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:03.008399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:03.076228 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40d708e-575e-41ea-af23-8dcb6854b880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:03.076869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40d708e-575e-41ea-af23-8dcb6854b880 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:03.077149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40d708e-575e-41ea-af23-8dcb6854b880 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:03.077423 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40d708e-575e-41ea-af23-8dcb6854b880 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:03.078149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40d708e-575e-41ea-af23-8dcb6854b880 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:03.078489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40d708e-575e-41ea-af23-8dcb6854b880 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:03.079930 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40d708e-575e-41ea-af23-8dcb6854b880 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:03.101617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 346/698] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:55:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:03.108969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40d708e-575e-41ea-af23-8dcb6854b880 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:03.137155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08459a5-c215-45f5-9e8c-52b45d60c4f3 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:03.175830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56784c0b-c186-4596-b514-e584c448bb33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:03.176707 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56784c0b-c186-4596-b514-e584c448bb33 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:03.177039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56784c0b-c186-4596-b514-e584c448bb33 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:03.177357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56784c0b-c186-4596-b514-e584c448bb33 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:03.178150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56784c0b-c186-4596-b514-e584c448bb33 None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=c11f1dd5a6b042c9b864a5861d0f3569) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:03.178451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56784c0b-c186-4596-b514-e584c448bb33 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:03.180006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56784c0b-c186-4596-b514-e584c448bb33 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:03.186983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 353/699] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:03 2022] GET /identity/v3/role_assignments?user.id=c11f1dd5a6b042c9b864a5861d0f3569&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 608 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:03.303674 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:03.307961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40d708e-575e-41ea-af23-8dcb6854b880 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:03.358354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af901fb-cfee-489c-bec3-002b0cfb612b None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:03.358689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af901fb-cfee-489c-bec3-002b0cfb612b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:03.358918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af901fb-cfee-489c-bec3-002b0cfb612b None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:03.361776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 354/700] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:03.362706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68382e70-50fd-4945-af44-95ffde978d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:03.363394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68382e70-50fd-4945-af44-95ffde978d9e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:03.363683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68382e70-50fd-4945-af44-95ffde978d9e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:03.363959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68382e70-50fd-4945-af44-95ffde978d9e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:03.366298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9ae352-bef8-4624-be1d-b0edf5a58e11 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:03.366582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9ae352-bef8-4624-be1d-b0edf5a58e11 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:03.366852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9ae352-bef8-4624-be1d-b0edf5a58e11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:03.386606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-7a9ae352-bef8-4624-be1d-b0edf5a58e11 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:03.387866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7a9ae352-bef8-4624-be1d-b0edf5a58e11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:03.388170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68382e70-50fd-4945-af44-95ffde978d9e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:03.389661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68382e70-50fd-4945-af44-95ffde978d9e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-68382e70-50fd-4945-af44-95ffde978d9e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:03.420760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:03.421769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:03.422636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:03.422636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:03.422636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:03.422636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:03.422636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 347/701] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:03.428744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68382e70-50fd-4945-af44-95ffde978d9e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:03.463837 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:03.473387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 355/702] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:55:03.496363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9ae352-bef8-4624-be1d-b0edf5a58e11 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:03.557737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68382e70-50fd-4945-af44-95ffde978d9e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:03.600715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ec0b1e-30bd-4cfd-b8fa-f709fcbf170e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:03.601338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ec0b1e-30bd-4cfd-b8fa-f709fcbf170e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:03.601611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ec0b1e-30bd-4cfd-b8fa-f709fcbf170e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:03.601883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ec0b1e-30bd-4cfd-b8fa-f709fcbf170e None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:03.602583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ec0b1e-30bd-4cfd-b8fa-f709fcbf170e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:03.602895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ec0b1e-30bd-4cfd-b8fa-f709fcbf170e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:03.604310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ec0b1e-30bd-4cfd-b8fa-f709fcbf170e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:03.619793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 348/703] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:03.625957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ec0b1e-30bd-4cfd-b8fa-f709fcbf170e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:03.644661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:03.655466 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9ae352-bef8-4624-be1d-b0edf5a58e11 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:03.705500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c99f54-f587-4ffa-bf32-a325e62df670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:03.708059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c99f54-f587-4ffa-bf32-a325e62df670 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:03.708547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c99f54-f587-4ffa-bf32-a325e62df670 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:03.708949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c99f54-f587-4ffa-bf32-a325e62df670 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:03.709790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c99f54-f587-4ffa-bf32-a325e62df670 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:03.711805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c99f54-f587-4ffa-bf32-a325e62df670 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-a0c99f54-f587-4ffa-bf32-a325e62df670 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:03.727580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:03.728775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 31 00:55:03.729848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:03.732766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 356/704] 10.222.0.42 () {56 vars in 1129 bytes} [Sun Jul 31 00:55:03 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:03.738143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c99f54-f587-4ffa-bf32-a325e62df670 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:03.780080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ec0b1e-30bd-4cfd-b8fa-f709fcbf170e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:03.819852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:03.830307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6037a412-6925-4591-9b10-f3729e31d093 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:03.831135 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6037a412-6925-4591-9b10-f3729e31d093 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:03.831577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6037a412-6925-4591-9b10-f3729e31d093 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:03.831961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6037a412-6925-4591-9b10-f3729e31d093 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:03.840843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6037a412-6925-4591-9b10-f3729e31d093 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:03.843379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6037a412-6925-4591-9b10-f3729e31d093 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-6037a412-6925-4591-9b10-f3729e31d093 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:03.866472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:03.867710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:03.868887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:03.868887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:03.868887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:03.868887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:03.868887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:03.868887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:03.868887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:03.868887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:03.868887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:55:03.868887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:03.870673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 349/705] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Jul 31 00:55:03 2022] GET /identity/v3/users/nova => generated 82 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:03.875947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c99f54-f587-4ffa-bf32-a325e62df670 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:03.876752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6037a412-6925-4591-9b10-f3729e31d093 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:03.920875 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e2cd2b-b260-49d8-b2bc-ba19d695cec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:03.921767 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e2cd2b-b260-49d8-b2bc-ba19d695cec5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:03.922573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e2cd2b-b260-49d8-b2bc-ba19d695cec5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:03.922992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e2cd2b-b260-49d8-b2bc-ba19d695cec5 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:03.924413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e2cd2b-b260-49d8-b2bc-ba19d695cec5 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:03.924896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e2cd2b-b260-49d8-b2bc-ba19d695cec5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:03.926981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e2cd2b-b260-49d8-b2bc-ba19d695cec5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:03.933633 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 357/706] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Jul 31 00:55:03 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:03.939871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e2cd2b-b260-49d8-b2bc-ba19d695cec5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:03.983912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:04.018112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6037a412-6925-4591-9b10-f3729e31d093 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:04.046470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e2cd2b-b260-49d8-b2bc-ba19d695cec5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:04.054197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b667d54b-5ccd-4da4-9b1c-ad3b95feb1ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:04.054943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b667d54b-5ccd-4da4-9b1c-ad3b95feb1ec None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:04.055324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b667d54b-5ccd-4da4-9b1c-ad3b95feb1ec None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:04.055731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b667d54b-5ccd-4da4-9b1c-ad3b95feb1ec None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:04.056648 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b667d54b-5ccd-4da4-9b1c-ad3b95feb1ec None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:04.057057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b667d54b-5ccd-4da4-9b1c-ad3b95feb1ec None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:04.058890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b667d54b-5ccd-4da4-9b1c-ad3b95feb1ec None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:04.086196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121dbfb6-5c6b-4472-bccb-bcd15faa84d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:04.087179 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121dbfb6-5c6b-4472-bccb-bcd15faa84d7 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:04.087687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121dbfb6-5c6b-4472-bccb-bcd15faa84d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:04.088154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121dbfb6-5c6b-4472-bccb-bcd15faa84d7 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:04.089027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121dbfb6-5c6b-4472-bccb-bcd15faa84d7 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:04.089538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121dbfb6-5c6b-4472-bccb-bcd15faa84d7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:04.091117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121dbfb6-5c6b-4472-bccb-bcd15faa84d7 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:04.094084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 350/707] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Jul 31 00:55:03 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:04.097950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 358/708] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Jul 31 00:55:03 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:04.103500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:04.103702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b667d54b-5ccd-4da4-9b1c-ad3b95feb1ec None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:04.203384 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a531fb-6063-4ba1-b434-7a215f5bb5ac None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:04.203748 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a531fb-6063-4ba1-b434-7a215f5bb5ac None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:04.203917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a531fb-6063-4ba1-b434-7a215f5bb5ac None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:04.205261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 359/709] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:04.212240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942fb539-f611-4c96-a9bc-d862ac3c73a8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:04.212537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942fb539-f611-4c96-a9bc-d862ac3c73a8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:04.212787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942fb539-f611-4c96-a9bc-d862ac3c73a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:04.245956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-942fb539-f611-4c96-a9bc-d862ac3c73a8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:04.247275 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-942fb539-f611-4c96-a9bc-d862ac3c73a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:04.290253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b667d54b-5ccd-4da4-9b1c-ad3b95feb1ec None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:04.336835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:04.344982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 360/710] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:55:04.346568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75de3562-9888-45e1-986a-de5d7d57914c None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:04.346850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75de3562-9888-45e1-986a-de5d7d57914c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:04.347126 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75de3562-9888-45e1-986a-de5d7d57914c None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:04.352389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 361/711] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:04.352560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156aff9e-6450-4963-a1a6-0597fabf294d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:04.352881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156aff9e-6450-4963-a1a6-0597fabf294d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:04.353152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156aff9e-6450-4963-a1a6-0597fabf294d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:04.353421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156aff9e-6450-4963-a1a6-0597fabf294d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:04.358145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4db58d-3e3a-4cca-8ad1-a1cd1c3fc244 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:04.358515 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4db58d-3e3a-4cca-8ad1-a1cd1c3fc244 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:04.358693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4db58d-3e3a-4cca-8ad1-a1cd1c3fc244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:04.388791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-de4db58d-3e3a-4cca-8ad1-a1cd1c3fc244 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:04.390135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-de4db58d-3e3a-4cca-8ad1-a1cd1c3fc244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:04.456265 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156aff9e-6450-4963-a1a6-0597fabf294d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:04.457851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156aff9e-6450-4963-a1a6-0597fabf294d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:04.464292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 362/712] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:55:04.465274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4db58d-3e3a-4cca-8ad1-a1cd1c3fc244 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:04.517200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:04.570325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4db58d-3e3a-4cca-8ad1-a1cd1c3fc244 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:04.571015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 351/713] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 472 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:55:04.571909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156aff9e-6450-4963-a1a6-0597fabf294d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:04.610419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:04.621230 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e95530-4b79-4bd0-a8a8-b009ad390926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:04.622090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e95530-4b79-4bd0-a8a8-b009ad390926 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:04.622521 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e95530-4b79-4bd0-a8a8-b009ad390926 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:04.622972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e95530-4b79-4bd0-a8a8-b009ad390926 None admin] PATH_INFO: `/v3/services/network` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:04.624378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e95530-4b79-4bd0-a8a8-b009ad390926 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:04.626425 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e95530-4b79-4bd0-a8a8-b009ad390926 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-d2e95530-4b79-4bd0-a8a8-b009ad390926 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:04.638792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:04.640062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 31 00:55:04.641162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:04.644233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 363/714] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:04 2022] GET /identity/v3/services/network => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:04.645870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e95530-4b79-4bd0-a8a8-b009ad390926 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:04.708991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156aff9e-6450-4963-a1a6-0597fabf294d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:04.756729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f5a896-7944-46ca-add1-f02c45bfbc56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:04.756951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e95530-4b79-4bd0-a8a8-b009ad390926 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:04.758097 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f5a896-7944-46ca-add1-f02c45bfbc56 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:04.758845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f5a896-7944-46ca-add1-f02c45bfbc56 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:04.759297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f5a896-7944-46ca-add1-f02c45bfbc56 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:04.774421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f5a896-7944-46ca-add1-f02c45bfbc56 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:04.775967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-72f5a896-7944-46ca-add1-f02c45bfbc56 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:04.783999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:04.785238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:04.786262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:04.786262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:04.786262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:04.786262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:04.786262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:04.786262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:04.786262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:04.786262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:04.786262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:04.786262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:04.786262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:55:04.786262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:04.789087 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 352/715] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:55:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:04.790132 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f5a896-7944-46ca-add1-f02c45bfbc56 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:04.830450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797e6d98-9f37-4886-81fb-baa80691f17e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:04.831527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797e6d98-9f37-4886-81fb-baa80691f17e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:04.832134 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797e6d98-9f37-4886-81fb-baa80691f17e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:04.832599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797e6d98-9f37-4886-81fb-baa80691f17e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:04.852559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797e6d98-9f37-4886-81fb-baa80691f17e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:04.857494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797e6d98-9f37-4886-81fb-baa80691f17e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-797e6d98-9f37-4886-81fb-baa80691f17e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:04.888295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:04.889389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:04.890476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:04.890476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:04.890476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:04.890476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:04.898989 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 364/716] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:04.900561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797e6d98-9f37-4886-81fb-baa80691f17e None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:04.938959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:04.952259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f5a896-7944-46ca-add1-f02c45bfbc56 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:04.998866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad188c4f-f9e0-4f16-9d16-03c097d2e0a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:05.000456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad188c4f-f9e0-4f16-9d16-03c097d2e0a6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:05.000927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad188c4f-f9e0-4f16-9d16-03c097d2e0a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:05.001353 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad188c4f-f9e0-4f16-9d16-03c097d2e0a6 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:05.002203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad188c4f-f9e0-4f16-9d16-03c097d2e0a6 None admin] RBAC: Adding query filter params (name=network) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:05.002663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad188c4f-f9e0-4f16-9d16-03c097d2e0a6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:05.004910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad188c4f-f9e0-4f16-9d16-03c097d2e0a6 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:05.011417 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 353/717] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:04 2022] GET /identity/v3/services?name=network => generated 124 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:05.012706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad188c4f-f9e0-4f16-9d16-03c097d2e0a6 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:05.113814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797e6d98-9f37-4886-81fb-baa80691f17e None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:05.157922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4424d832-ebdc-4439-a21c-dd4ad4f7c9b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:05.158827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4424d832-ebdc-4439-a21c-dd4ad4f7c9b2 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:05.159258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4424d832-ebdc-4439-a21c-dd4ad4f7c9b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:05.159712 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4424d832-ebdc-4439-a21c-dd4ad4f7c9b2 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:05.160578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4424d832-ebdc-4439-a21c-dd4ad4f7c9b2 None admin] RBAC: Adding query filter params (name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:05.161046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4424d832-ebdc-4439-a21c-dd4ad4f7c9b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:05.161405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad188c4f-f9e0-4f16-9d16-03c097d2e0a6 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:05.162752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4424d832-ebdc-4439-a21c-dd4ad4f7c9b2 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:05.172923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 365/718] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:55:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:05.173953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4424d832-ebdc-4439-a21c-dd4ad4f7c9b2 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:05.220144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:05.242104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ff9e42-47e0-4f8a-8ef8-ae05578d63a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:05.242824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ff9e42-47e0-4f8a-8ef8-ae05578d63a6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:05.243168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ff9e42-47e0-4f8a-8ef8-ae05578d63a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:05.243536 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ff9e42-47e0-4f8a-8ef8-ae05578d63a6 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:05.244611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ff9e42-47e0-4f8a-8ef8-ae05578d63a6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:05.245615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ff9e42-47e0-4f8a-8ef8-ae05578d63a6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:05.247408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ff9e42-47e0-4f8a-8ef8-ae05578d63a6 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:05.267442 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 354/719] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:05.268623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ff9e42-47e0-4f8a-8ef8-ae05578d63a6 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:05.291561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:05.313496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4424d832-ebdc-4439-a21c-dd4ad4f7c9b2 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:05.353937 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb23e8a-064e-4ea1-a9d1-3af88b0825c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:05.354846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb23e8a-064e-4ea1-a9d1-3af88b0825c3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:05.355276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb23e8a-064e-4ea1-a9d1-3af88b0825c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:05.355766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb23e8a-064e-4ea1-a9d1-3af88b0825c3 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:05.356737 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb23e8a-064e-4ea1-a9d1-3af88b0825c3 None admin] RBAC: Adding query filter params (type=network) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:05.357257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb23e8a-064e-4ea1-a9d1-3af88b0825c3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:05.358773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb23e8a-064e-4ea1-a9d1-3af88b0825c3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:05.365095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 366/720] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:05 2022] GET /identity/v3/services?type=network => generated 124 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:05.366324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb23e8a-064e-4ea1-a9d1-3af88b0825c3 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:05.416214 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ff9e42-47e0-4f8a-8ef8-ae05578d63a6 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:05.474336 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc33d607-cb8b-4f01-af6c-3196881ca9b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:05.476814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc33d607-cb8b-4f01-af6c-3196881ca9b6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:05.477568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc33d607-cb8b-4f01-af6c-3196881ca9b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:05.478799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc33d607-cb8b-4f01-af6c-3196881ca9b6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:05.495611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:05.508634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc33d607-cb8b-4f01-af6c-3196881ca9b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:05.510601 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc33d607-cb8b-4f01-af6c-3196881ca9b6 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:05.513832 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb23e8a-064e-4ea1-a9d1-3af88b0825c3 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-fc33d607-cb8b-4f01-af6c-3196881ca9b6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:05.542209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:05.543220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:05.544351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:05.544351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:05.544351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:05.544351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:05.556948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 355/721] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:05.562378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc33d607-cb8b-4f01-af6c-3196881ca9b6 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:05.566399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0587f4-b7e0-44f8-a715-a3830ebdb079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:05.567494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0587f4-b7e0-44f8-a715-a3830ebdb079 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:05.568006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0587f4-b7e0-44f8-a715-a3830ebdb079 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:05.568537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0587f4-b7e0-44f8-a715-a3830ebdb079 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:05.580556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0587f4-b7e0-44f8-a715-a3830ebdb079 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:05.582803 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0587f4-b7e0-44f8-a715-a3830ebdb079 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-fe0587f4-b7e0-44f8-a715-a3830ebdb079 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:05.598705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:05.599744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:05.601203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:05.601203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:05.601203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:05.601203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:05.601203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:05.601203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:05.601203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:05.601203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:05.604800 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 367/722] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:05.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0587f4-b7e0-44f8-a715-a3830ebdb079 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:05.678881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:05.719280 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc33d607-cb8b-4f01-af6c-3196881ca9b6 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:05.738138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0587f4-b7e0-44f8-a715-a3830ebdb079 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:05.755090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b07a6ec-1103-4bf9-a304-dff2941cc0db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:05.755748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b07a6ec-1103-4bf9-a304-dff2941cc0db None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:05.756029 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b07a6ec-1103-4bf9-a304-dff2941cc0db None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:05.756636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b07a6ec-1103-4bf9-a304-dff2941cc0db None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:05.756984 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b07a6ec-1103-4bf9-a304-dff2941cc0db None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:05.757286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b07a6ec-1103-4bf9-a304-dff2941cc0db None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:05.758652 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b07a6ec-1103-4bf9-a304-dff2941cc0db None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:05.772483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 356/723] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:05.777858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b07a6ec-1103-4bf9-a304-dff2941cc0db None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:05.780250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a987e1ef-bb9f-4a9f-ba3c-4ad652558a7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:05.781578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a987e1ef-bb9f-4a9f-ba3c-4ad652558a7e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:05.782266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a987e1ef-bb9f-4a9f-ba3c-4ad652558a7e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:05.782745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a987e1ef-bb9f-4a9f-ba3c-4ad652558a7e None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:05.783746 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a987e1ef-bb9f-4a9f-ba3c-4ad652558a7e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:05.784265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a987e1ef-bb9f-4a9f-ba3c-4ad652558a7e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:05.786189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a987e1ef-bb9f-4a9f-ba3c-4ad652558a7e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:05.802562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 368/724] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:05.809030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a987e1ef-bb9f-4a9f-ba3c-4ad652558a7e None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:05.825385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:05.884256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b07a6ec-1103-4bf9-a304-dff2941cc0db None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:05.916167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:05.928678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b3304b-aa8d-4b80-8047-22f6e3e78a2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:05.929349 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b3304b-aa8d-4b80-8047-22f6e3e78a2d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:05.929577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b3304b-aa8d-4b80-8047-22f6e3e78a2d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:05.929952 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b3304b-aa8d-4b80-8047-22f6e3e78a2d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:05.938416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b3304b-aa8d-4b80-8047-22f6e3e78a2d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:05.940052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b3304b-aa8d-4b80-8047-22f6e3e78a2d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:05.943088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a987e1ef-bb9f-4a9f-ba3c-4ad652558a7e None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-02b3304b-aa8d-4b80-8047-22f6e3e78a2d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:05.952971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:05.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:05.954941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:05.954941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:05.954941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:05.954941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:05.954941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:05.954941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:05.954941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:05.954941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:05.954941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 31 00:55:05.954941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:05.954941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 357/725] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Jul 31 00:55:05 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:05.960789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6713a6d5-9edc-478c-b185-531bc8b16ab2 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:05.961139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6713a6d5-9edc-478c-b185-531bc8b16ab2 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:05.961406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6713a6d5-9edc-478c-b185-531bc8b16ab2 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:05.963184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 358/726] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:05 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:05.964920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6713a6d5-9edc-478c-b185-531bc8b16ab2 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:05.990187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050066ea-93ce-479e-b605-bacd95f22ccd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:05.991302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050066ea-93ce-479e-b605-bacd95f22ccd None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:05.991824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050066ea-93ce-479e-b605-bacd95f22ccd None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:05.992300 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050066ea-93ce-479e-b605-bacd95f22ccd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:05.993294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050066ea-93ce-479e-b605-bacd95f22ccd None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=5b0bd572ae7a4c7f9e07fe0b9708aa7e) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:05.993832 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050066ea-93ce-479e-b605-bacd95f22ccd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:05.995710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050066ea-93ce-479e-b605-bacd95f22ccd None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:06.002935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 369/727] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:05 2022] GET /identity/v3/role_assignments?user.id=5b0bd572ae7a4c7f9e07fe0b9708aa7e&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:06.008949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c868355d-6c22-4139-a774-821c282afd41 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:06.009895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c868355d-6c22-4139-a774-821c282afd41 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:06.010177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c868355d-6c22-4139-a774-821c282afd41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:06.058662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-c868355d-6c22-4139-a774-821c282afd41 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:06.059925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c868355d-6c22-4139-a774-821c282afd41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:06.129951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6713a6d5-9edc-478c-b185-531bc8b16ab2 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:06.169713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d15322b-dcb9-4434-92ff-56931861774a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:06.170397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d15322b-dcb9-4434-92ff-56931861774a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:06.170678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d15322b-dcb9-4434-92ff-56931861774a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:06.170954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d15322b-dcb9-4434-92ff-56931861774a None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:06.172489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d15322b-dcb9-4434-92ff-56931861774a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:06.172811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d15322b-dcb9-4434-92ff-56931861774a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:06.174358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d15322b-dcb9-4434-92ff-56931861774a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:06.196548 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 359/728] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Jul 31 00:55:05 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:06.202452 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d15322b-dcb9-4434-92ff-56931861774a None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:06.209862 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 370/729] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:55:06.248879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c868355d-6c22-4139-a774-821c282afd41 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:06.252069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:06.256577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:06.318842 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d15322b-dcb9-4434-92ff-56931861774a None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:06.442635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c868355d-6c22-4139-a774-821c282afd41 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:06.493207 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368ca817-9875-4fe9-a6ae-0bc089aea304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:06.493858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368ca817-9875-4fe9-a6ae-0bc089aea304 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:06.494073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368ca817-9875-4fe9-a6ae-0bc089aea304 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:06.494688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368ca817-9875-4fe9-a6ae-0bc089aea304 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:06.551714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7abb4ec-5159-4e38-bb2d-5c1e3d22b71b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:06.552377 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7abb4ec-5159-4e38-bb2d-5c1e3d22b71b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:06.552658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7abb4ec-5159-4e38-bb2d-5c1e3d22b71b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:06.552933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7abb4ec-5159-4e38-bb2d-5c1e3d22b71b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:06.566341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7abb4ec-5159-4e38-bb2d-5c1e3d22b71b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-d7abb4ec-5159-4e38-bb2d-5c1e3d22b71b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:06.570981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:06.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:06.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:06.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:06.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:06.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:06.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:06.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:06.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:06.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:06.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:06.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:06.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:06.577234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 371/730] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:06.579118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e855022-8c96-47d5-8f47-7ff8fdf1b227 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:06.579421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e855022-8c96-47d5-8f47-7ff8fdf1b227 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:06.579701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e855022-8c96-47d5-8f47-7ff8fdf1b227 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:06.581566 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 372/731] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:06.582611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e855022-8c96-47d5-8f47-7ff8fdf1b227 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:06.591139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368ca817-9875-4fe9-a6ae-0bc089aea304 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:06.592862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368ca817-9875-4fe9-a6ae-0bc089aea304 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:06.651008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:06.667146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:06.712102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e855022-8c96-47d5-8f47-7ff8fdf1b227 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:06.716259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 360/732] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 514 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:55:06.717817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ce7752-a92f-4ffd-92d9-e99429002afe None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:06.718248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ce7752-a92f-4ffd-92d9-e99429002afe None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:06.719172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ce7752-a92f-4ffd-92d9-e99429002afe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:06.739205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b9ce7752-a92f-4ffd-92d9-e99429002afe None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:06.740512 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b9ce7752-a92f-4ffd-92d9-e99429002afe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:06.753436 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc19b1e-a6f3-4fab-b53c-52ff093dfbaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:06.754358 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc19b1e-a6f3-4fab-b53c-52ff093dfbaa None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:06.754832 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc19b1e-a6f3-4fab-b53c-52ff093dfbaa None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:06.755283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc19b1e-a6f3-4fab-b53c-52ff093dfbaa None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:06.756318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc19b1e-a6f3-4fab-b53c-52ff093dfbaa None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:06.756805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc19b1e-a6f3-4fab-b53c-52ff093dfbaa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:06.758434 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc19b1e-a6f3-4fab-b53c-52ff093dfbaa None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:06.771554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 373/733] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:55:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:06.772824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc19b1e-a6f3-4fab-b53c-52ff093dfbaa None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:06.816283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 361/734] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:55:06.824344 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ce7752-a92f-4ffd-92d9-e99429002afe None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:06.888842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc19b1e-a6f3-4fab-b53c-52ff093dfbaa None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:06.921174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:06.928015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:06.931130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52853f2-b251-4dd4-bc92-8337c63cec1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:06.932119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52853f2-b251-4dd4-bc92-8337c63cec1f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:06.932739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52853f2-b251-4dd4-bc92-8337c63cec1f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:06.933291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52853f2-b251-4dd4-bc92-8337c63cec1f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:06.945789 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52853f2-b251-4dd4-bc92-8337c63cec1f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:06.947681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52853f2-b251-4dd4-bc92-8337c63cec1f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:06.961459 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ce7752-a92f-4ffd-92d9-e99429002afe None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-a52853f2-b251-4dd4-bc92-8337c63cec1f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:06.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:06.964997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:06.966012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:06.966012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:06.966012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:06.966012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:06.966012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 374/735] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:06.966816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52853f2-b251-4dd4-bc92-8337c63cec1f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:07.016424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b217e5-9dd9-49a8-a504-89b9057cf109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:07.017544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b217e5-9dd9-49a8-a504-89b9057cf109 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:07.018715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b217e5-9dd9-49a8-a504-89b9057cf109 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:07.019214 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b217e5-9dd9-49a8-a504-89b9057cf109 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:07.038786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b217e5-9dd9-49a8-a504-89b9057cf109 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:07.040438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b217e5-9dd9-49a8-a504-89b9057cf109 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-81b217e5-9dd9-49a8-a504-89b9057cf109 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:07.058998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:07.059970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:07.060864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:07.060864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:07.060864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:07.060864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:07.060864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 362/736] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:07.061596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b217e5-9dd9-49a8-a504-89b9057cf109 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:07.111140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52853f2-b251-4dd4-bc92-8337c63cec1f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:07.152879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facb357f-6ea0-47cb-8900-aba167c5aaee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:07.152879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facb357f-6ea0-47cb-8900-aba167c5aaee None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:07.153189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facb357f-6ea0-47cb-8900-aba167c5aaee None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:07.153486 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facb357f-6ea0-47cb-8900-aba167c5aaee None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:07.154194 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facb357f-6ea0-47cb-8900-aba167c5aaee None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:07.155677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facb357f-6ea0-47cb-8900-aba167c5aaee None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:07.183632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 375/737] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:55:06 2022] POST /identity/v3/services => generated 251 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:07.183632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facb357f-6ea0-47cb-8900-aba167c5aaee None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:07.210422 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b217e5-9dd9-49a8-a504-89b9057cf109 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:07.309058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:07.364783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:07.374715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1034e295-f017-4538-b343-aac193067b1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:07.383395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1034e295-f017-4538-b343-aac193067b1c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:07.383395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1034e295-f017-4538-b343-aac193067b1c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:07.383395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1034e295-f017-4538-b343-aac193067b1c None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:07.383395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1034e295-f017-4538-b343-aac193067b1c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:07.383395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1034e295-f017-4538-b343-aac193067b1c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:07.383395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1034e295-f017-4538-b343-aac193067b1c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:07.412134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 363/738] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:07.419483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1034e295-f017-4538-b343-aac193067b1c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:07.464283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facb357f-6ea0-47cb-8900-aba167c5aaee None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:07.531975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c29d04a-73a4-42df-9dee-e4c0af27c779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:07.532573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c29d04a-73a4-42df-9dee-e4c0af27c779 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:07.533577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c29d04a-73a4-42df-9dee-e4c0af27c779 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:07.533577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c29d04a-73a4-42df-9dee-e4c0af27c779 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:07.534423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c29d04a-73a4-42df-9dee-e4c0af27c779 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:07.536081 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c29d04a-73a4-42df-9dee-e4c0af27c779 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:07.538724 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c29d04a-73a4-42df-9dee-e4c0af27c779 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:07.567654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 376/739] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 386 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:07.570780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c29d04a-73a4-42df-9dee-e4c0af27c779 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:07.658013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1034e295-f017-4538-b343-aac193067b1c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:07.751753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1598276f-a21a-429c-a6ec-ba4771f23514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:07.753526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1598276f-a21a-429c-a6ec-ba4771f23514 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:07.754195 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1598276f-a21a-429c-a6ec-ba4771f23514 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:07.756098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1598276f-a21a-429c-a6ec-ba4771f23514 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:07.776335 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1598276f-a21a-429c-a6ec-ba4771f23514 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:07.785639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1598276f-a21a-429c-a6ec-ba4771f23514 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-1598276f-a21a-429c-a6ec-ba4771f23514 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:07.819051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:07.819996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:07.820871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:07.820871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:07.820871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:07.820871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:07.820871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:07.820871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:07.820871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:07.820871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:07.832403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 364/740] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 413 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:07.835778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:07.839783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1598276f-a21a-429c-a6ec-ba4771f23514 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:07.845307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c29d04a-73a4-42df-9dee-e4c0af27c779 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:08.028343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71be2de-c32f-4710-9a9b-6b5a982cc780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:08.029305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:08.030516 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71be2de-c32f-4710-9a9b-6b5a982cc780 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:08.032258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71be2de-c32f-4710-9a9b-6b5a982cc780 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:08.032996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71be2de-c32f-4710-9a9b-6b5a982cc780 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:08.053738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71be2de-c32f-4710-9a9b-6b5a982cc780 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:08.056791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71be2de-c32f-4710-9a9b-6b5a982cc780 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-b71be2de-c32f-4710-9a9b-6b5a982cc780 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:08.097220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:08.098295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:08.099227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:08.099227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:08.099227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:08.099227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:08.099227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:08.099227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:08.099227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:08.099227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:08.099227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 31 00:55:08.099227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:08.105364 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 377/741] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:07 2022] GET /identity/v3/users/swift => generated 83 bytes in 528 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:08.106351 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71be2de-c32f-4710-9a9b-6b5a982cc780 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:08.136300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1598276f-a21a-429c-a6ec-ba4771f23514 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:08.180689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf8cf27-e594-4397-9e7c-62f5a51896d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:08.181312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf8cf27-e594-4397-9e7c-62f5a51896d6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:08.181759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf8cf27-e594-4397-9e7c-62f5a51896d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:08.182036 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf8cf27-e594-4397-9e7c-62f5a51896d6 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:08.183166 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf8cf27-e594-4397-9e7c-62f5a51896d6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:08.183804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf8cf27-e594-4397-9e7c-62f5a51896d6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:08.185473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf8cf27-e594-4397-9e7c-62f5a51896d6 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:08.202081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 365/742] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 363 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:08.204062 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e447f0-0c0b-418d-bd49-2133aa11140c None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:08.204350 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e447f0-0c0b-418d-bd49-2133aa11140c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:08.204626 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e447f0-0c0b-418d-bd49-2133aa11140c None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:08.206192 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 366/743] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:08.209392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e447f0-0c0b-418d-bd49-2133aa11140c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:08.235435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71be2de-c32f-4710-9a9b-6b5a982cc780 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:08.272117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eed8fcd-15b7-4063-bef3-518544cfab92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:08.272976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eed8fcd-15b7-4063-bef3-518544cfab92 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:08.273329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eed8fcd-15b7-4063-bef3-518544cfab92 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:08.273669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eed8fcd-15b7-4063-bef3-518544cfab92 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:08.274577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eed8fcd-15b7-4063-bef3-518544cfab92 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:08.274982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eed8fcd-15b7-4063-bef3-518544cfab92 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:08.276665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eed8fcd-15b7-4063-bef3-518544cfab92 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:08.295670 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 378/744] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Jul 31 00:55:08 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:08.297192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20405f3e-73a6-4e49-a781-ae89f1dd125e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:08.297577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20405f3e-73a6-4e49-a781-ae89f1dd125e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:08.297953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20405f3e-73a6-4e49-a781-ae89f1dd125e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:08.317579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-20405f3e-73a6-4e49-a781-ae89f1dd125e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:08.318908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-20405f3e-73a6-4e49-a781-ae89f1dd125e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:08.332912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:08.342485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e447f0-0c0b-418d-bd49-2133aa11140c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:08.344183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:08.384228 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194fe36d-4ab9-4f1a-b2a4-04f8f4c21e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:08.384751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194fe36d-4ab9-4f1a-b2a4-04f8f4c21e45 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:08.385035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194fe36d-4ab9-4f1a-b2a4-04f8f4c21e45 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:08.385304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194fe36d-4ab9-4f1a-b2a4-04f8f4c21e45 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:08.386080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194fe36d-4ab9-4f1a-b2a4-04f8f4c21e45 None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=8cdffe91785e42d798b931082969598e) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:08.386451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194fe36d-4ab9-4f1a-b2a4-04f8f4c21e45 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:08.388007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194fe36d-4ab9-4f1a-b2a4-04f8f4c21e45 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:08.399549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 367/745] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:08 2022] GET /identity/v3/role_assignments?user.id=8cdffe91785e42d798b931082969598e&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:08.399549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194fe36d-4ab9-4f1a-b2a4-04f8f4c21e45 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:08.402021 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 379/746] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:08 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 31 00:55:08.403745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f0ab44-2d78-43a4-af2c-d7f5ec2fbaa9 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:08.404364 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f0ab44-2d78-43a4-af2c-d7f5ec2fbaa9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:08.404787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f0ab44-2d78-43a4-af2c-d7f5ec2fbaa9 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:08.406339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 380/747] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:08.417764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cf547b-600c-43df-8889-1cbabef33760 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:08.418480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cf547b-600c-43df-8889-1cbabef33760 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:08.418965 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cf547b-600c-43df-8889-1cbabef33760 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:08.443757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a3cf547b-600c-43df-8889-1cbabef33760 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:08.445340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a3cf547b-600c-43df-8889-1cbabef33760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:08.522537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 381/748] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:08 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 31 00:55:08.528359 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cf547b-600c-43df-8889-1cbabef33760 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:08.552814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194fe36d-4ab9-4f1a-b2a4-04f8f4c21e45 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:08.594273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea52a4a-74c3-42fe-9b0f-209428076f5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:08.595165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea52a4a-74c3-42fe-9b0f-209428076f5f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:08.595477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea52a4a-74c3-42fe-9b0f-209428076f5f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:08.595758 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea52a4a-74c3-42fe-9b0f-209428076f5f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:08.626363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:08.650855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cf547b-600c-43df-8889-1cbabef33760 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:08.658796 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:08.690633 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a13be26-0dce-43f5-89e9-d3b7369f7c86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:08.690851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea52a4a-74c3-42fe-9b0f-209428076f5f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:08.691754 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a13be26-0dce-43f5-89e9-d3b7369f7c86 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:08.692113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea52a4a-74c3-42fe-9b0f-209428076f5f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:08.692371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a13be26-0dce-43f5-89e9-d3b7369f7c86 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:08.692781 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a13be26-0dce-43f5-89e9-d3b7369f7c86 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:08.693604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a13be26-0dce-43f5-89e9-d3b7369f7c86 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:08.695288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a13be26-0dce-43f5-89e9-d3b7369f7c86 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:08.719753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 382/749] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:55:08 2022] POST /identity/v3/services => generated 240 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:08.721102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a13be26-0dce-43f5-89e9-d3b7369f7c86 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:08.867797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 368/750] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:08 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 31 00:55:08.879388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea52a4a-74c3-42fe-9b0f-209428076f5f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:08.956428 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a13be26-0dce-43f5-89e9-d3b7369f7c86 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:08.983055 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:08.998278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:09.010645 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6df4be4-ca5f-41e5-b107-c9bcac253574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:09.011311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6df4be4-ca5f-41e5-b107-c9bcac253574 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:09.011621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6df4be4-ca5f-41e5-b107-c9bcac253574 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:09.011914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6df4be4-ca5f-41e5-b107-c9bcac253574 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:09.029350 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6df4be4-ca5f-41e5-b107-c9bcac253574 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-f6df4be4-ca5f-41e5-b107-c9bcac253574 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:09.036468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:09.037430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:09.038245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:09.038245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:09.038245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:09.038245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:09.038245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:09.038245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:09.038245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:09.038245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:09.038245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:09.038245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:09.038245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:09.038245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:09.039442 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea52a4a-74c3-42fe-9b0f-209428076f5f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:09.041322 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 383/751] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:09.047901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6df4be4-ca5f-41e5-b107-c9bcac253574 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:09.082075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f5c9b3-6d69-4745-9db5-d9dea65ab3db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:09.082727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f5c9b3-6d69-4745-9db5-d9dea65ab3db None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:09.083007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f5c9b3-6d69-4745-9db5-d9dea65ab3db None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:09.083280 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f5c9b3-6d69-4745-9db5-d9dea65ab3db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:09.096664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f5c9b3-6d69-4745-9db5-d9dea65ab3db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:09.098578 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f5c9b3-6d69-4745-9db5-d9dea65ab3db None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-14f5c9b3-6d69-4745-9db5-d9dea65ab3db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:09.115343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:09.116312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:09.117196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:09.117196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:09.117196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:09.117196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:09.117196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 369/752] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:09.122930 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f5c9b3-6d69-4745-9db5-d9dea65ab3db None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:09.171736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6df4be4-ca5f-41e5-b107-c9bcac253574 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:09.213024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410652e4-55e2-43cf-85a7-3f838f8ce56b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:09.213910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410652e4-55e2-43cf-85a7-3f838f8ce56b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:09.214339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410652e4-55e2-43cf-85a7-3f838f8ce56b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:09.214758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410652e4-55e2-43cf-85a7-3f838f8ce56b None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:09.215654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410652e4-55e2-43cf-85a7-3f838f8ce56b None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:09.216153 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410652e4-55e2-43cf-85a7-3f838f8ce56b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:09.217836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410652e4-55e2-43cf-85a7-3f838f8ce56b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:09.229190 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 384/753] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:55:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:09.237051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410652e4-55e2-43cf-85a7-3f838f8ce56b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:09.326388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f5c9b3-6d69-4745-9db5-d9dea65ab3db None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:09.395173 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:09.401210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:09.436817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56d6654-27d2-4b62-b5bb-30c105f51945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:09.438277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56d6654-27d2-4b62-b5bb-30c105f51945 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:09.438708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56d6654-27d2-4b62-b5bb-30c105f51945 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:09.439128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56d6654-27d2-4b62-b5bb-30c105f51945 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:09.440001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56d6654-27d2-4b62-b5bb-30c105f51945 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:09.440478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56d6654-27d2-4b62-b5bb-30c105f51945 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:09.442004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56d6654-27d2-4b62-b5bb-30c105f51945 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:09.473757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 370/754] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:09.477834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410652e4-55e2-43cf-85a7-3f838f8ce56b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:09.480415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56d6654-27d2-4b62-b5bb-30c105f51945 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:09.515191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef76729-1749-4aee-b22b-bbe1c6c98c0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:09.516154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef76729-1749-4aee-b22b-bbe1c6c98c0c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:09.516611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef76729-1749-4aee-b22b-bbe1c6c98c0c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:09.517100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef76729-1749-4aee-b22b-bbe1c6c98c0c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:09.529313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef76729-1749-4aee-b22b-bbe1c6c98c0c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:09.531070 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef76729-1749-4aee-b22b-bbe1c6c98c0c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-eef76729-1749-4aee-b22b-bbe1c6c98c0c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:09.548123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:09.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:09.550383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:09.550383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:09.550383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:09.550383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:09.552506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 385/755] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:09.558373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef76729-1749-4aee-b22b-bbe1c6c98c0c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:09.674868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56d6654-27d2-4b62-b5bb-30c105f51945 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:09.714244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4ff11e-7928-4a3a-bebd-4715b527ddac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:09.715015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4ff11e-7928-4a3a-bebd-4715b527ddac None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:09.715204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4ff11e-7928-4a3a-bebd-4715b527ddac None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:09.715519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4ff11e-7928-4a3a-bebd-4715b527ddac None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:09.726066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef76729-1749-4aee-b22b-bbe1c6c98c0c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:09.732766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4ff11e-7928-4a3a-bebd-4715b527ddac None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:09.735226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4ff11e-7928-4a3a-bebd-4715b527ddac None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:09.742060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-8a4ff11e-7928-4a3a-bebd-4715b527ddac None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:09.760073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:09.761059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:09.762073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:09.762073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:09.762073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:09.762073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:09.762073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:09.762073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:09.762073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:09.762073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:09.766299 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 371/756] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:09.771658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50205ae-87c5-48d1-8766-17c73d64ab3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:09.772427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4ff11e-7928-4a3a-bebd-4715b527ddac None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:09.772757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50205ae-87c5-48d1-8766-17c73d64ab3e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:09.773298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50205ae-87c5-48d1-8766-17c73d64ab3e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:09.773757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50205ae-87c5-48d1-8766-17c73d64ab3e None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:09.774704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50205ae-87c5-48d1-8766-17c73d64ab3e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:09.775221 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50205ae-87c5-48d1-8766-17c73d64ab3e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:09.778097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50205ae-87c5-48d1-8766-17c73d64ab3e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:09.794126 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 386/757] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:09.800276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135889c5-47cd-4594-bfa0-4b0de4a8aa35 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:09.801168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135889c5-47cd-4594-bfa0-4b0de4a8aa35 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:09.801647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135889c5-47cd-4594-bfa0-4b0de4a8aa35 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:09.803645 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 387/758] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:09 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:09.806349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135889c5-47cd-4594-bfa0-4b0de4a8aa35 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:09.835829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:09.905411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4ff11e-7928-4a3a-bebd-4715b527ddac None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:09.931700 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135889c5-47cd-4594-bfa0-4b0de4a8aa35 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:09.943782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b1bd78-24df-4c14-838f-475366863185 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:09.944555 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b1bd78-24df-4c14-838f-475366863185 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:09.944940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b1bd78-24df-4c14-838f-475366863185 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:09.945307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b1bd78-24df-4c14-838f-475366863185 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:09.946262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b1bd78-24df-4c14-838f-475366863185 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:09.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b1bd78-24df-4c14-838f-475366863185 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:09.948243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b1bd78-24df-4c14-838f-475366863185 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:09.962945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 372/759] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:09.964541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82c0f4f-47b6-4721-a56a-7cd23ea986f3 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:09.964924 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82c0f4f-47b6-4721-a56a-7cd23ea986f3 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:09.965290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82c0f4f-47b6-4721-a56a-7cd23ea986f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:09.981266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e9a1a8-5c40-4ecc-8c50-26a83c89a1d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:09.982375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e9a1a8-5c40-4ecc-8c50-26a83c89a1d2 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:09.982804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e9a1a8-5c40-4ecc-8c50-26a83c89a1d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:09.983223 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e9a1a8-5c40-4ecc-8c50-26a83c89a1d2 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:09.986021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d82c0f4f-47b6-4721-a56a-7cd23ea986f3 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:09.987734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d82c0f4f-47b6-4721-a56a-7cd23ea986f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:09.992910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e9a1a8-5c40-4ecc-8c50-26a83c89a1d2 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:09.994655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e9a1a8-5c40-4ecc-8c50-26a83c89a1d2 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-f6e9a1a8-5c40-4ecc-8c50-26a83c89a1d2 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:10.009112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:10.010273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:10.011290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:10.011290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:10.011290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:10.011290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:10.011290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:10.011290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:10.011290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:10.011290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:10.011290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:55:10.011290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:10.018474 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 388/760] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Jul 31 00:55:09 2022] GET /identity/v3/users/nova => generated 82 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:10.019873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e9a1a8-5c40-4ecc-8c50-26a83c89a1d2 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:10.057256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:10.071752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 373/761] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:09 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:10.072843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82c0f4f-47b6-4721-a56a-7cd23ea986f3 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:10.119068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:10.144635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e9a1a8-5c40-4ecc-8c50-26a83c89a1d2 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:10.184669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f276d030-1bdc-47ed-8a9d-45e950a79f6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:10.185690 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f276d030-1bdc-47ed-8a9d-45e950a79f6c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:10.186116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f276d030-1bdc-47ed-8a9d-45e950a79f6c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:10.186560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f276d030-1bdc-47ed-8a9d-45e950a79f6c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:10.188187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f276d030-1bdc-47ed-8a9d-45e950a79f6c None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=c11f1dd5a6b042c9b864a5861d0f3569) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:10.188669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f276d030-1bdc-47ed-8a9d-45e950a79f6c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:10.190369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f276d030-1bdc-47ed-8a9d-45e950a79f6c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:10.197338 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 389/762] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:10 2022] GET /identity/v3/role_assignments?user.id=c11f1dd5a6b042c9b864a5861d0f3569&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:10.198383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f276d030-1bdc-47ed-8a9d-45e950a79f6c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:10.215907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82c0f4f-47b6-4721-a56a-7cd23ea986f3 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:10.314695 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4383df50-ef3d-458d-b00e-a2fe7dc0662a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:10.315252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4383df50-ef3d-458d-b00e-a2fe7dc0662a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:10.315570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4383df50-ef3d-458d-b00e-a2fe7dc0662a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:10.315839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4383df50-ef3d-458d-b00e-a2fe7dc0662a None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:10.316641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4383df50-ef3d-458d-b00e-a2fe7dc0662a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:10.321227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4383df50-ef3d-458d-b00e-a2fe7dc0662a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:10.322805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4383df50-ef3d-458d-b00e-a2fe7dc0662a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:10.363702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 374/763] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Jul 31 00:55:10 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:10.367380 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f276d030-1bdc-47ed-8a9d-45e950a79f6c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:10.367717 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:10.371924 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4383df50-ef3d-458d-b00e-a2fe7dc0662a None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:10.413608 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0a8b62-b828-48d3-8730-219d90ef08c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:10.414374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0a8b62-b828-48d3-8730-219d90ef08c9 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:10.414724 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0a8b62-b828-48d3-8730-219d90ef08c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:10.415069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0a8b62-b828-48d3-8730-219d90ef08c9 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:10.415811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0a8b62-b828-48d3-8730-219d90ef08c9 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:10.417280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0a8b62-b828-48d3-8730-219d90ef08c9 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-9a0a8b62-b828-48d3-8730-219d90ef08c9 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:10.428145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:10.429150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:10.429972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 390/764] 10.222.0.42 () {56 vars in 1129 bytes} [Sun Jul 31 00:55:10 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:10.438792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0a8b62-b828-48d3-8730-219d90ef08c9 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:10.576945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:10.594428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4383df50-ef3d-458d-b00e-a2fe7dc0662a None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:10.638610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa54f5d-9cc0-4e91-a96c-fc55369e6afe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:10.639819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa54f5d-9cc0-4e91-a96c-fc55369e6afe None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:10.640374 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa54f5d-9cc0-4e91-a96c-fc55369e6afe None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:10.640852 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa54f5d-9cc0-4e91-a96c-fc55369e6afe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:10.654357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa54f5d-9cc0-4e91-a96c-fc55369e6afe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:10.656996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa54f5d-9cc0-4e91-a96c-fc55369e6afe None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:10.661741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0a8b62-b828-48d3-8730-219d90ef08c9 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-2aa54f5d-9cc0-4e91-a96c-fc55369e6afe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:10.678292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:10.679617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:10.680691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:10.680691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:10.680691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:10.680691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:10.683113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 375/765] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:10.689858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa54f5d-9cc0-4e91-a96c-fc55369e6afe None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:10.805872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa54f5d-9cc0-4e91-a96c-fc55369e6afe None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:10.817310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653273ff-0936-4a84-a654-c4e71ea86921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:10.817962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653273ff-0936-4a84-a654-c4e71ea86921 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:10.818243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653273ff-0936-4a84-a654-c4e71ea86921 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:10.818518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653273ff-0936-4a84-a654-c4e71ea86921 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:10.819662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653273ff-0936-4a84-a654-c4e71ea86921 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:10.820004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653273ff-0936-4a84-a654-c4e71ea86921 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:10.821535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653273ff-0936-4a84-a654-c4e71ea86921 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:10.853453 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 391/766] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Jul 31 00:55:10 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 412 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:10.854768 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c067ae5d-6839-4182-980a-eef434585ff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:10.855728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c067ae5d-6839-4182-980a-eef434585ff6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:10.856227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c067ae5d-6839-4182-980a-eef434585ff6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:10.856698 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c067ae5d-6839-4182-980a-eef434585ff6 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:10.857762 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c067ae5d-6839-4182-980a-eef434585ff6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:10.858297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c067ae5d-6839-4182-980a-eef434585ff6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:10.860075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c067ae5d-6839-4182-980a-eef434585ff6 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:10.868932 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653273ff-0936-4a84-a654-c4e71ea86921 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:10.878262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 376/767] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:10.879765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:10.884502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c067ae5d-6839-4182-980a-eef434585ff6 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:10.952492 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:11.031953 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c067ae5d-6839-4182-980a-eef434585ff6 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:11.071303 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653273ff-0936-4a84-a654-c4e71ea86921 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:11.091654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a998a71-ce5a-43f0-ad42-c6a2861946c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:11.092683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a998a71-ce5a-43f0-ad42-c6a2861946c8 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:11.092965 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a998a71-ce5a-43f0-ad42-c6a2861946c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:11.093241 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a998a71-ce5a-43f0-ad42-c6a2861946c8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:11.103420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a998a71-ce5a-43f0-ad42-c6a2861946c8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:11.105227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a998a71-ce5a-43f0-ad42-c6a2861946c8 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:11.113781 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f99c577-358e-4767-a436-8e426c083cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:11.114556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f99c577-358e-4767-a436-8e426c083cfc None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:11.114960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f99c577-358e-4767-a436-8e426c083cfc None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:11.115332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f99c577-358e-4767-a436-8e426c083cfc None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:11.116164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f99c577-358e-4767-a436-8e426c083cfc None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:11.116568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f99c577-358e-4767-a436-8e426c083cfc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:11.118123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f99c577-358e-4767-a436-8e426c083cfc None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-6a998a71-ce5a-43f0-ad42-c6a2861946c8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:11.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:11.120155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:11.121609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:11.121609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:11.121609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:11.121609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:11.121609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:11.121609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:11.121609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:11.121609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:11.121609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 377/768] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:11.127390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c704e0e3-b3e7-4066-aef7-2c4567387293 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:11.127390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c704e0e3-b3e7-4066-aef7-2c4567387293 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:11.127390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c704e0e3-b3e7-4066-aef7-2c4567387293 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:11.127853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 392/770] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Jul 31 00:55:10 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:11.128740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f99c577-358e-4767-a436-8e426c083cfc None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:11.130411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 378/770] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:11.132657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c94ba8-5b87-4458-8f75-521e2d435192 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:11.132978 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c94ba8-5b87-4458-8f75-521e2d435192 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:11.133574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c94ba8-5b87-4458-8f75-521e2d435192 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:11.157816 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-49c94ba8-5b87-4458-8f75-521e2d435192 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:11.159341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-49c94ba8-5b87-4458-8f75-521e2d435192 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:11.233948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:11.238517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:11.267758 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 379/771] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:11 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:11.267758 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c94ba8-5b87-4458-8f75-521e2d435192 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:11.272470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f99c577-358e-4767-a436-8e426c083cfc None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:11.325731 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b03e22-030b-4717-b137-70bb31766edf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:11.326392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b03e22-030b-4717-b137-70bb31766edf None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:11.326665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b03e22-030b-4717-b137-70bb31766edf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:11.326933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b03e22-030b-4717-b137-70bb31766edf None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:11.327726 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b03e22-030b-4717-b137-70bb31766edf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:11.328619 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b03e22-030b-4717-b137-70bb31766edf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:11.330067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b03e22-030b-4717-b137-70bb31766edf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:11.347411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 393/772] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:11.347411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b03e22-030b-4717-b137-70bb31766edf None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:11.415824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c94ba8-5b87-4458-8f75-521e2d435192 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:11.469537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bb6f3d-064f-4288-aaf9-2f591cee3664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:11.470184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bb6f3d-064f-4288-aaf9-2f591cee3664 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:11.470467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bb6f3d-064f-4288-aaf9-2f591cee3664 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:11.470743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bb6f3d-064f-4288-aaf9-2f591cee3664 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:11.471661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bb6f3d-064f-4288-aaf9-2f591cee3664 None admin] RBAC: Adding query filter params (interface=public, service_id=680fd7a11d8c4c1d8c6e99a94fc9e9f8, region_id=RegionOne) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:11.471960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bb6f3d-064f-4288-aaf9-2f591cee3664 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:11.473347 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bb6f3d-064f-4288-aaf9-2f591cee3664 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:11.481870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 380/773] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Jul 31 00:55:11 2022] GET /identity/v3/endpoints?service_id=680fd7a11d8c4c1d8c6e99a94fc9e9f8&interface=public®ion_id=RegionOne => generated 194 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:11.483263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58a630f-dbe8-476c-b56c-b0c0faf279d3 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:11.483639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58a630f-dbe8-476c-b56c-b0c0faf279d3 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:11.483978 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58a630f-dbe8-476c-b56c-b0c0faf279d3 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:11.485798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b03e22-030b-4717-b137-70bb31766edf None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:11.486126 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 381/774] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:11.487140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58a630f-dbe8-476c-b56c-b0c0faf279d3 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:11.543005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6369b97a-3fa3-48f5-9c62-f8bd7a729bcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:11.544070 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6369b97a-3fa3-48f5-9c62-f8bd7a729bcb None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:11.544383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6369b97a-3fa3-48f5-9c62-f8bd7a729bcb None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:11.544661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6369b97a-3fa3-48f5-9c62-f8bd7a729bcb None admin] PATH_INFO: `/v3/roles/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:11.557639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6369b97a-3fa3-48f5-9c62-f8bd7a729bcb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-6369b97a-3fa3-48f5-9c62-f8bd7a729bcb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:11.563229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:11.564344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:11.565261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 394/775] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:55:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:11.567059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:11.568289 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6369b97a-3fa3-48f5-9c62-f8bd7a729bcb None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:11.572772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:11.637517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58a630f-dbe8-476c-b56c-b0c0faf279d3 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:11.677239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44ea767-a058-4568-88df-7b489e53b173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:11.677806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44ea767-a058-4568-88df-7b489e53b173 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:11.678080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44ea767-a058-4568-88df-7b489e53b173 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:11.678360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44ea767-a058-4568-88df-7b489e53b173 None admin] PATH_INFO: `/v3/projects/8b41573c9f3146d19a306089d686015c/users/5b0bd572ae7a4c7f9e07fe0b9708aa7e/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:11.680405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44ea767-a058-4568-88df-7b489e53b173 None admin] RBAC: Authorizing `identity:create_grant(project_id=8b41573c9f3146d19a306089d686015c, user_id=5b0bd572ae7a4c7f9e07fe0b9708aa7e, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:11.681861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44ea767-a058-4568-88df-7b489e53b173 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:11.690333 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6369b97a-3fa3-48f5-9c62-f8bd7a729bcb None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:11.707385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 382/776] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:55:11 2022] PUT /identity/v3/projects/8b41573c9f3146d19a306089d686015c/users/5b0bd572ae7a4c7f9e07fe0b9708aa7e/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:55:11.707385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e3b58-6bc7-4c41-8edb-f1de36693903 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:11.707385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e3b58-6bc7-4c41-8edb-f1de36693903 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:11.707385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e3b58-6bc7-4c41-8edb-f1de36693903 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:11.728667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a92e3b58-6bc7-4c41-8edb-f1de36693903 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:11.730001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a92e3b58-6bc7-4c41-8edb-f1de36693903 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:11.731648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b8eabc-0b0f-481a-a772-e9c1f99914e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:11.732352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b8eabc-0b0f-481a-a772-e9c1f99914e3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:11.732638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b8eabc-0b0f-481a-a772-e9c1f99914e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:11.732962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b8eabc-0b0f-481a-a772-e9c1f99914e3 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:11.733640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b8eabc-0b0f-481a-a772-e9c1f99914e3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:11.735910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b8eabc-0b0f-481a-a772-e9c1f99914e3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:11.742465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 395/777] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Jul 31 00:55:11 2022] GET /identity/v3/services => generated 772 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:11.743542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b8eabc-0b0f-481a-a772-e9c1f99914e3 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:11.821937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 383/778] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:11 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:11.856827 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e3b58-6bc7-4c41-8edb-f1de36693903 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:11.867601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:11.938135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b8eabc-0b0f-481a-a772-e9c1f99914e3 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:11.943053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:11.980979 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02076f1-6559-446c-9c29-177fcdf5295d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:11.981635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02076f1-6559-446c-9c29-177fcdf5295d None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:11.981919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02076f1-6559-446c-9c29-177fcdf5295d None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:11.983557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02076f1-6559-446c-9c29-177fcdf5295d None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:11.983557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02076f1-6559-446c-9c29-177fcdf5295d None admin] RBAC: Adding query filter params (name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:11.983557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02076f1-6559-446c-9c29-177fcdf5295d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:11.986385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02076f1-6559-446c-9c29-177fcdf5295d None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:11.997355 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 396/779] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:55:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:12.003676 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02076f1-6559-446c-9c29-177fcdf5295d None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:12.016297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e3b58-6bc7-4c41-8edb-f1de36693903 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:12.063998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d62be73-ed70-40db-9e02-054b9f9c8e96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:12.065200 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62be73-ed70-40db-9e02-054b9f9c8e96 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:12.065484 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62be73-ed70-40db-9e02-054b9f9c8e96 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:12.065756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62be73-ed70-40db-9e02-054b9f9c8e96 None admin] PATH_INFO: `/v3/services/network` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:12.066657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d62be73-ed70-40db-9e02-054b9f9c8e96 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:12.068838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d62be73-ed70-40db-9e02-054b9f9c8e96 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-1d62be73-ed70-40db-9e02-054b9f9c8e96 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:12.087067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:12.088161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:12.089013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 384/780] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:11 2022] GET /identity/v3/services/network => generated 88 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:12.096681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d62be73-ed70-40db-9e02-054b9f9c8e96 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:12.141948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02076f1-6559-446c-9c29-177fcdf5295d None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:12.214828 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:12.229131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61a57ec-b6f1-4150-834b-83b67f7ff760 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:12.229781 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61a57ec-b6f1-4150-834b-83b67f7ff760 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:12.230058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61a57ec-b6f1-4150-834b-83b67f7ff760 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:12.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61a57ec-b6f1-4150-834b-83b67f7ff760 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:12.243484 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d62be73-ed70-40db-9e02-054b9f9c8e96 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:12.253425 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61a57ec-b6f1-4150-834b-83b67f7ff760 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:12.254888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61a57ec-b6f1-4150-834b-83b67f7ff760 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:12.256909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-e61a57ec-b6f1-4150-834b-83b67f7ff760 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:12.282764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:12.283777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:12.284648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:12.284648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:12.284648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:12.284648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:12.286930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 397/781] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:12.292362 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61a57ec-b6f1-4150-834b-83b67f7ff760 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:12.318663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158218a8-69d3-4ada-a70b-b5a6069d254b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:12.319781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158218a8-69d3-4ada-a70b-b5a6069d254b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:12.320174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158218a8-69d3-4ada-a70b-b5a6069d254b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:12.320650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158218a8-69d3-4ada-a70b-b5a6069d254b None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:12.321871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158218a8-69d3-4ada-a70b-b5a6069d254b None admin] RBAC: Adding query filter params (name=network) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:12.322756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158218a8-69d3-4ada-a70b-b5a6069d254b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:12.329713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158218a8-69d3-4ada-a70b-b5a6069d254b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:12.343028 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 385/782] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:12 2022] GET /identity/v3/services?name=network => generated 124 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:12.349801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158218a8-69d3-4ada-a70b-b5a6069d254b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:12.428095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61a57ec-b6f1-4150-834b-83b67f7ff760 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:12.487051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637481f2-d408-49d8-9f05-2af949d5525f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:12.487592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637481f2-d408-49d8-9f05-2af949d5525f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:12.487865 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637481f2-d408-49d8-9f05-2af949d5525f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:12.488134 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637481f2-d408-49d8-9f05-2af949d5525f None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:12.488820 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637481f2-d408-49d8-9f05-2af949d5525f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:12.489117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637481f2-d408-49d8-9f05-2af949d5525f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:12.490472 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637481f2-d408-49d8-9f05-2af949d5525f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:12.515039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 398/783] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:12.526062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637481f2-d408-49d8-9f05-2af949d5525f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:12.573157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158218a8-69d3-4ada-a70b-b5a6069d254b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:12.596855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:12.612545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcd946a-a741-48b9-a5c3-8d5adc49abee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:12.613480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcd946a-a741-48b9-a5c3-8d5adc49abee None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:12.613947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcd946a-a741-48b9-a5c3-8d5adc49abee None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:12.614451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcd946a-a741-48b9-a5c3-8d5adc49abee None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:12.615462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcd946a-a741-48b9-a5c3-8d5adc49abee None admin] RBAC: Adding query filter params (type=network) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:12.615942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcd946a-a741-48b9-a5c3-8d5adc49abee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:12.617657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcd946a-a741-48b9-a5c3-8d5adc49abee None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:12.629877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 386/784] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:12 2022] GET /identity/v3/services?type=network => generated 350 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:12.631645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a508bd-83bd-4567-8cea-6bde76112aee None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:12.632134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a508bd-83bd-4567-8cea-6bde76112aee None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:12.632585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a508bd-83bd-4567-8cea-6bde76112aee None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:12.634286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 387/785] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:12.635144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a508bd-83bd-4567-8cea-6bde76112aee None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:12.666439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637481f2-d408-49d8-9f05-2af949d5525f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:12.686071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:12.710873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0b7b58-1a0a-4c96-b34a-0c184c1761f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:12.711854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b7b58-1a0a-4c96-b34a-0c184c1761f3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:12.712365 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b7b58-1a0a-4c96-b34a-0c184c1761f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:12.712795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b7b58-1a0a-4c96-b34a-0c184c1761f3 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:12.723060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0b7b58-1a0a-4c96-b34a-0c184c1761f3 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:12.724872 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0b7b58-1a0a-4c96-b34a-0c184c1761f3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-4d0b7b58-1a0a-4c96-b34a-0c184c1761f3 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:12.740587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:12.741523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 399/786] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Jul 31 00:55:12 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5319ec-410c-4ceb-8964-07b315b678ee None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:12.747121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5319ec-410c-4ceb-8964-07b315b678ee None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:12.747855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5319ec-410c-4ceb-8964-07b315b678ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:12.768539 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ef5319ec-410c-4ceb-8964-07b315b678ee None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:12.770031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ef5319ec-410c-4ceb-8964-07b315b678ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:12.771725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a508bd-83bd-4567-8cea-6bde76112aee None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:12.807685 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00bdf5d-fb0d-498a-901c-2ad1af1a9d6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:12.808845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00bdf5d-fb0d-498a-901c-2ad1af1a9d6b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:12.809420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00bdf5d-fb0d-498a-901c-2ad1af1a9d6b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:12.809970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00bdf5d-fb0d-498a-901c-2ad1af1a9d6b None admin] PATH_INFO: `/v3/endpoints` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:12.811513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00bdf5d-fb0d-498a-901c-2ad1af1a9d6b None admin] RBAC: Adding query filter params (interface=public, service_id=9659d131e33742edaa67849babfa3fcc, region_id=RegionOne) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:12.812676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00bdf5d-fb0d-498a-901c-2ad1af1a9d6b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:12.815267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00bdf5d-fb0d-498a-901c-2ad1af1a9d6b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:12.826721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 388/787] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Jul 31 00:55:12 2022] GET /identity/v3/endpoints?service_id=9659d131e33742edaa67849babfa3fcc&interface=public®ion_id=RegionOne => generated 194 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:12.827783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00bdf5d-fb0d-498a-901c-2ad1af1a9d6b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:12.875226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 400/788] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:12 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:12.877149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5319ec-410c-4ceb-8964-07b315b678ee None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:12.943593 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:12.975823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:12.982817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00bdf5d-fb0d-498a-901c-2ad1af1a9d6b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:13.022573 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533a4f1b-53f5-4b3c-acbc-9fc3fda0bcce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:13.023372 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a4f1b-53f5-4b3c-acbc-9fc3fda0bcce None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:13.023708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a4f1b-53f5-4b3c-acbc-9fc3fda0bcce None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:13.023990 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a4f1b-53f5-4b3c-acbc-9fc3fda0bcce None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:13.024787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533a4f1b-53f5-4b3c-acbc-9fc3fda0bcce None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:13.025144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533a4f1b-53f5-4b3c-acbc-9fc3fda0bcce None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:13.026754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533a4f1b-53f5-4b3c-acbc-9fc3fda0bcce None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:13.047953 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 389/789] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Jul 31 00:55:12 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:13.048873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533a4f1b-53f5-4b3c-acbc-9fc3fda0bcce None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:13.049468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5319ec-410c-4ceb-8964-07b315b678ee None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:13.097283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579aace4-23b5-4aa9-8fec-44d07cba4e15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:13.098791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579aace4-23b5-4aa9-8fec-44d07cba4e15 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:13.099374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579aace4-23b5-4aa9-8fec-44d07cba4e15 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:13.099833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579aace4-23b5-4aa9-8fec-44d07cba4e15 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:13.100943 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-579aace4-23b5-4aa9-8fec-44d07cba4e15 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:13.103310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-579aace4-23b5-4aa9-8fec-44d07cba4e15 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:13.111147 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 401/790] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Jul 31 00:55:12 2022] GET /identity/v3/services => generated 772 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:13.112586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579aace4-23b5-4aa9-8fec-44d07cba4e15 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:13.174112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533a4f1b-53f5-4b3c-acbc-9fc3fda0bcce None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:13.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905f228a-fac3-4ae0-8e7e-29a7ba61bd98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:13.253225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905f228a-fac3-4ae0-8e7e-29a7ba61bd98 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:13.253684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905f228a-fac3-4ae0-8e7e-29a7ba61bd98 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:13.254324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905f228a-fac3-4ae0-8e7e-29a7ba61bd98 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:13.277755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905f228a-fac3-4ae0-8e7e-29a7ba61bd98 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:13.281579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-905f228a-fac3-4ae0-8e7e-29a7ba61bd98 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:13.288308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:13.289314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:13.290124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:13.290124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:13.290124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:13.290124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:13.290124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:13.290124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:13.290124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:13.290124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:13.290124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:13.290124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:13.290124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:13.290124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:13.292590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 390/791] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:13.300332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905f228a-fac3-4ae0-8e7e-29a7ba61bd98 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:13.306023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579aace4-23b5-4aa9-8fec-44d07cba4e15 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:13.337582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:13.366756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b6188c-c099-46eb-9d09-cd687d96c7c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:13.367948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b6188c-c099-46eb-9d09-cd687d96c7c4 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:13.368427 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b6188c-c099-46eb-9d09-cd687d96c7c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:13.368903 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b6188c-c099-46eb-9d09-cd687d96c7c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:13.387208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b6188c-c099-46eb-9d09-cd687d96c7c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:13.388415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b6188c-c099-46eb-9d09-cd687d96c7c4 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-a8b6188c-c099-46eb-9d09-cd687d96c7c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:13.407308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:13.408234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:13.409104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:13.409104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:13.409104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:13.409104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:13.411348 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 402/792] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:13.417125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b6188c-c099-46eb-9d09-cd687d96c7c4 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:13.453980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905f228a-fac3-4ae0-8e7e-29a7ba61bd98 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:13.493711 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c7fb9d-269e-4db7-99de-14dbff2cdeef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:13.494332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c7fb9d-269e-4db7-99de-14dbff2cdeef None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:13.494610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c7fb9d-269e-4db7-99de-14dbff2cdeef None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:13.494887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c7fb9d-269e-4db7-99de-14dbff2cdeef None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:13.495618 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c7fb9d-269e-4db7-99de-14dbff2cdeef None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:13.495961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c7fb9d-269e-4db7-99de-14dbff2cdeef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:13.497430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c7fb9d-269e-4db7-99de-14dbff2cdeef None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:13.516014 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 391/793] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:55:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:13.524263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c7fb9d-269e-4db7-99de-14dbff2cdeef None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:13.630883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:13.646280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b6188c-c099-46eb-9d09-cd687d96c7c4 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:13.655245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c7fb9d-269e-4db7-99de-14dbff2cdeef None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:13.695689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e879a98-a393-45a2-8389-233ee76927ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:13.696332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e879a98-a393-45a2-8389-233ee76927ed None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:13.696617 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e879a98-a393-45a2-8389-233ee76927ed None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:13.696893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e879a98-a393-45a2-8389-233ee76927ed None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:13.697602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e879a98-a393-45a2-8389-233ee76927ed None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:13.697909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e879a98-a393-45a2-8389-233ee76927ed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:13.699288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e879a98-a393-45a2-8389-233ee76927ed None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:13.714353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 403/794] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:13.721603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e879a98-a393-45a2-8389-233ee76927ed None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:13.723380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c2184c-fde7-48bd-8ea8-ba7f7f9f6d81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:13.724051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c2184c-fde7-48bd-8ea8-ba7f7f9f6d81 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:13.724329 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c2184c-fde7-48bd-8ea8-ba7f7f9f6d81 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:13.724602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c2184c-fde7-48bd-8ea8-ba7f7f9f6d81 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:13.736802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c2184c-fde7-48bd-8ea8-ba7f7f9f6d81 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:13.738385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c2184c-fde7-48bd-8ea8-ba7f7f9f6d81 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:13.743815 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-16c2184c-fde7-48bd-8ea8-ba7f7f9f6d81 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:13.768367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:13.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:13.770162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:13.770162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:13.770162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:13.770162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:13.774168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 392/795] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:13.782514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c2184c-fde7-48bd-8ea8-ba7f7f9f6d81 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:13.869381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e879a98-a393-45a2-8389-233ee76927ed None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:13.914397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5135ca55-c9e9-40bc-8217-720c3480ecd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:13.915323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5135ca55-c9e9-40bc-8217-720c3480ecd8 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:13.915635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5135ca55-c9e9-40bc-8217-720c3480ecd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:13.916160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5135ca55-c9e9-40bc-8217-720c3480ecd8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:13.926882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5135ca55-c9e9-40bc-8217-720c3480ecd8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:13.928366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5135ca55-c9e9-40bc-8217-720c3480ecd8 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-5135ca55-c9e9-40bc-8217-720c3480ecd8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:13.944240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:13.946297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:13.948373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:13.948373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:13.948373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:13.948373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:13.948373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:13.948373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:13.948373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:13.948373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:13.950867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 404/796] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:13.952150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5135ca55-c9e9-40bc-8217-720c3480ecd8 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:13.997405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c2184c-fde7-48bd-8ea8-ba7f7f9f6d81 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:14.055781 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:14.076470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5135ca55-c9e9-40bc-8217-720c3480ecd8 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:14.092942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:14.103920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9240e6c4-6a12-41a1-bef1-860be9d67a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:14.104550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240e6c4-6a12-41a1-bef1-860be9d67a36 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:14.104833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240e6c4-6a12-41a1-bef1-860be9d67a36 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:14.105150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240e6c4-6a12-41a1-bef1-860be9d67a36 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:14.105850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9240e6c4-6a12-41a1-bef1-860be9d67a36 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:14.111800 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9240e6c4-6a12-41a1-bef1-860be9d67a36 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:14.113193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9240e6c4-6a12-41a1-bef1-860be9d67a36 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:14.121447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85115e01-1137-4b8d-a631-da2057fe0baa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:14.122448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85115e01-1137-4b8d-a631-da2057fe0baa None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:14.122878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85115e01-1137-4b8d-a631-da2057fe0baa None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:14.123299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85115e01-1137-4b8d-a631-da2057fe0baa None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:14.124968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85115e01-1137-4b8d-a631-da2057fe0baa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:14.125457 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85115e01-1137-4b8d-a631-da2057fe0baa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:14.127059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85115e01-1137-4b8d-a631-da2057fe0baa None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:14.145477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 393/797] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 359 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:14.146137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 405/798] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:14.152506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9240e6c4-6a12-41a1-bef1-860be9d67a36 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:14.152668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85115e01-1137-4b8d-a631-da2057fe0baa None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:14.275938 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85115e01-1137-4b8d-a631-da2057fe0baa None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:14.305444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:14.315709 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9559c15-0122-4e86-b158-34597be42eba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:14.316485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9559c15-0122-4e86-b158-34597be42eba None admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:14.316770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9559c15-0122-4e86-b158-34597be42eba None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:14.317043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9559c15-0122-4e86-b158-34597be42eba None admin] PATH_INFO: `/v3/projects/8b41573c9f3146d19a306089d686015c/users/8cdffe91785e42d798b931082969598e/roles/26925162cb82467f981eae19826e00c9` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:14.325413 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9240e6c4-6a12-41a1-bef1-860be9d67a36 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:14.333733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9559c15-0122-4e86-b158-34597be42eba None admin] RBAC: Authorizing `identity:create_grant(project_id=8b41573c9f3146d19a306089d686015c, user_id=8cdffe91785e42d798b931082969598e, role_id=26925162cb82467f981eae19826e00c9)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:14.335196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9559c15-0122-4e86-b158-34597be42eba None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:14.353914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 406/799] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:55:14 2022] PUT /identity/v3/projects/8b41573c9f3146d19a306089d686015c/users/8cdffe91785e42d798b931082969598e/roles/26925162cb82467f981eae19826e00c9 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 00:55:14.355937 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43947617-0e49-4225-875b-7ec00d03b8c9 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:14.356497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43947617-0e49-4225-875b-7ec00d03b8c9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:14.356774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43947617-0e49-4225-875b-7ec00d03b8c9 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:14.363538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 407/800] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:14.365733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43c0eee-3e39-4527-b0d4-2a51e3c2bc60 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:14.366782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43c0eee-3e39-4527-b0d4-2a51e3c2bc60 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:14.366782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43c0eee-3e39-4527-b0d4-2a51e3c2bc60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:14.380057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442e7d3e-e562-41b7-a9da-380026daec51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:14.380649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442e7d3e-e562-41b7-a9da-380026daec51 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:14.380927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442e7d3e-e562-41b7-a9da-380026daec51 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:14.381197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442e7d3e-e562-41b7-a9da-380026daec51 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:14.387346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a43c0eee-3e39-4527-b0d4-2a51e3c2bc60 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:14.388638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a43c0eee-3e39-4527-b0d4-2a51e3c2bc60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:14.394489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442e7d3e-e562-41b7-a9da-380026daec51 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:14.396463 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442e7d3e-e562-41b7-a9da-380026daec51 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-442e7d3e-e562-41b7-a9da-380026daec51 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:14.414606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:14.415838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:14.417034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:14.417034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:14.417034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:14.417034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:14.417034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:14.417034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:14.417034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:14.417034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:14.417034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 31 00:55:14.417034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:14.420012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 394/801] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:14 2022] GET /identity/v3/users/swift => generated 83 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:14.422713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c09f6e4-9622-439d-853e-c0c1b268ce53 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:14.423274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c09f6e4-9622-439d-853e-c0c1b268ce53 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:14.423713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c09f6e4-9622-439d-853e-c0c1b268ce53 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:14.426240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 395/802] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:14.427145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c09f6e4-9622-439d-853e-c0c1b268ce53 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:14.487498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 408/803] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:14 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:14.489354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c34981c-256a-44ca-b30b-0976b6fcf388 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:14.489835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c34981c-256a-44ca-b30b-0976b6fcf388 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:14.490278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c34981c-256a-44ca-b30b-0976b6fcf388 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:14.490691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:14.528491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-9c34981c-256a-44ca-b30b-0976b6fcf388 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:14.529705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9c34981c-256a-44ca-b30b-0976b6fcf388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:14.562421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c09f6e4-9622-439d-853e-c0c1b268ce53 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:14.606296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba85aa3-ae2d-4cc7-ba81-10624063640e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:14.607784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba85aa3-ae2d-4cc7-ba81-10624063640e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:14.608279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba85aa3-ae2d-4cc7-ba81-10624063640e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:14.608737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba85aa3-ae2d-4cc7-ba81-10624063640e None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:14.609768 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba85aa3-ae2d-4cc7-ba81-10624063640e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:14.610311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba85aa3-ae2d-4cc7-ba81-10624063640e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:14.613175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba85aa3-ae2d-4cc7-ba81-10624063640e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:14.630301 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 409/804] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:14 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:14.636670 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c34981c-256a-44ca-b30b-0976b6fcf388 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:14.639139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 396/805] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Jul 31 00:55:14 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:14.646633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba85aa3-ae2d-4cc7-ba81-10624063640e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:14.694007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:14.766993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba85aa3-ae2d-4cc7-ba81-10624063640e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:14.777780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:14.795988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c34981c-256a-44ca-b30b-0976b6fcf388 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:14.806286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b6d9df-4220-4dcf-9f77-39696596001f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:14.807090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b6d9df-4220-4dcf-9f77-39696596001f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:14.807523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b6d9df-4220-4dcf-9f77-39696596001f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:14.807904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b6d9df-4220-4dcf-9f77-39696596001f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:14.819701 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b6d9df-4220-4dcf-9f77-39696596001f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:14.821451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b6d9df-4220-4dcf-9f77-39696596001f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:14.839068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650f95f2-e828-4d3a-8904-9ce86f6d1edd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-02b6d9df-4220-4dcf-9f77-39696596001f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:14.839270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:14.840312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:14.841387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:14.841387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:14.841387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:14.841387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:14.841655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650f95f2-e828-4d3a-8904-9ce86f6d1edd None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:14.841655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650f95f2-e828-4d3a-8904-9ce86f6d1edd None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:14.841655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650f95f2-e828-4d3a-8904-9ce86f6d1edd None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:14.841655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650f95f2-e828-4d3a-8904-9ce86f6d1edd None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:14.844102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650f95f2-e828-4d3a-8904-9ce86f6d1edd None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:14.844255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 397/806] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:14.844255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b6d9df-4220-4dcf-9f77-39696596001f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-650f95f2-e828-4d3a-8904-9ce86f6d1edd None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:14.860454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:14.861614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 31 00:55:14.862632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:14.866742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 410/807] 10.222.0.42 () {56 vars in 1129 bytes} [Sun Jul 31 00:55:14 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:14.866742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650f95f2-e828-4d3a-8904-9ce86f6d1edd None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:14.966606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b6d9df-4220-4dcf-9f77-39696596001f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:14.987120 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:14.991064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650f95f2-e828-4d3a-8904-9ce86f6d1edd None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:15.004151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a4d5e1-7cce-4004-a88a-3456f0f43584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:15.004794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a4d5e1-7cce-4004-a88a-3456f0f43584 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:15.005074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a4d5e1-7cce-4004-a88a-3456f0f43584 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:15.005340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a4d5e1-7cce-4004-a88a-3456f0f43584 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:15.017387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a4d5e1-7cce-4004-a88a-3456f0f43584 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-36a4d5e1-7cce-4004-a88a-3456f0f43584 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:15.023348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:15.025436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 398/808] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:15.027961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a4d5e1-7cce-4004-a88a-3456f0f43584 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:15.029820 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146f4ce8-a1b2-43e8-8f66-d94b14a11131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:15.030070 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146f4ce8-a1b2-43e8-8f66-d94b14a11131 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:15.039388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146f4ce8-a1b2-43e8-8f66-d94b14a11131 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:15.039388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146f4ce8-a1b2-43e8-8f66-d94b14a11131 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:15.039388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146f4ce8-a1b2-43e8-8f66-d94b14a11131 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:15.039388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146f4ce8-a1b2-43e8-8f66-d94b14a11131 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:15.039388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146f4ce8-a1b2-43e8-8f66-d94b14a11131 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:15.047298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 411/809] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:15.051997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:15.052933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146f4ce8-a1b2-43e8-8f66-d94b14a11131 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:15.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a4d5e1-7cce-4004-a88a-3456f0f43584 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:15.164883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146f4ce8-a1b2-43e8-8f66-d94b14a11131 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:15.177965 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c38584-95a2-47ca-91f6-fd3459bf1f1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:15.178582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c38584-95a2-47ca-91f6-fd3459bf1f1a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:15.178905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c38584-95a2-47ca-91f6-fd3459bf1f1a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:15.179181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c38584-95a2-47ca-91f6-fd3459bf1f1a None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:15.180003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c38584-95a2-47ca-91f6-fd3459bf1f1a None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:15.180303 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c38584-95a2-47ca-91f6-fd3459bf1f1a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:15.181785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c38584-95a2-47ca-91f6-fd3459bf1f1a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:15.186824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 399/810] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Jul 31 00:55:15 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:15.192370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c38584-95a2-47ca-91f6-fd3459bf1f1a None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:15.202399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949e6fe9-b2fd-4822-a37f-29820ba6d1a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:15.203628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949e6fe9-b2fd-4822-a37f-29820ba6d1a5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:15.204006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949e6fe9-b2fd-4822-a37f-29820ba6d1a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:15.204285 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949e6fe9-b2fd-4822-a37f-29820ba6d1a5 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:15.204998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949e6fe9-b2fd-4822-a37f-29820ba6d1a5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:15.205303 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949e6fe9-b2fd-4822-a37f-29820ba6d1a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:15.206755 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949e6fe9-b2fd-4822-a37f-29820ba6d1a5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:15.216332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 412/811] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:55:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:15.217314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949e6fe9-b2fd-4822-a37f-29820ba6d1a5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:15.249249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:15.291302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:15.310256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c38584-95a2-47ca-91f6-fd3459bf1f1a None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:15.333943 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949e6fe9-b2fd-4822-a37f-29820ba6d1a5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:15.348094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a21dd6f-d1f2-4f87-9e21-b430d7b77044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:15.349055 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a21dd6f-d1f2-4f87-9e21-b430d7b77044 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:15.349476 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a21dd6f-d1f2-4f87-9e21-b430d7b77044 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:15.349922 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a21dd6f-d1f2-4f87-9e21-b430d7b77044 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:15.357447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a21dd6f-d1f2-4f87-9e21-b430d7b77044 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:15.359781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a21dd6f-d1f2-4f87-9e21-b430d7b77044 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:15.370169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61acf3d-a2a4-4c6a-9dd3-015456dcf95b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:15.370787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61acf3d-a2a4-4c6a-9dd3-015456dcf95b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:15.371063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61acf3d-a2a4-4c6a-9dd3-015456dcf95b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:15.371332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61acf3d-a2a4-4c6a-9dd3-015456dcf95b None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:15.372037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61acf3d-a2a4-4c6a-9dd3-015456dcf95b None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:15.372340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61acf3d-a2a4-4c6a-9dd3-015456dcf95b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-1a21dd6f-d1f2-4f87-9e21-b430d7b77044 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:15.373411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:15.374441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:15.376091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:15.376091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:15.376091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:15.376091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:15.376091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:15.376091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:15.376091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:15.376091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:15.376540 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61acf3d-a2a4-4c6a-9dd3-015456dcf95b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:15.378416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 400/812] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:15.379369 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a21dd6f-d1f2-4f87-9e21-b430d7b77044 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:15.384771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 413/813] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Jul 31 00:55:15 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:15.385680 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61acf3d-a2a4-4c6a-9dd3-015456dcf95b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:15.484818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:15.492930 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a21dd6f-d1f2-4f87-9e21-b430d7b77044 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:15.505823 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61acf3d-a2a4-4c6a-9dd3-015456dcf95b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:15.538792 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c40ecb-b621-4b05-a72d-6009f382d216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:15.539547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c40ecb-b621-4b05-a72d-6009f382d216 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:15.539860 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c40ecb-b621-4b05-a72d-6009f382d216 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:15.540134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c40ecb-b621-4b05-a72d-6009f382d216 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:15.543982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedd4ab0-ae6e-497c-9dcb-c6e533d61222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:15.544998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedd4ab0-ae6e-497c-9dcb-c6e533d61222 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:15.545420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedd4ab0-ae6e-497c-9dcb-c6e533d61222 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:15.545835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedd4ab0-ae6e-497c-9dcb-c6e533d61222 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:15.547304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedd4ab0-ae6e-497c-9dcb-c6e533d61222 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:15.547775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedd4ab0-ae6e-497c-9dcb-c6e533d61222 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:15.549612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedd4ab0-ae6e-497c-9dcb-c6e533d61222 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:15.557027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c40ecb-b621-4b05-a72d-6009f382d216 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:15.559016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c40ecb-b621-4b05-a72d-6009f382d216 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:15.563323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:15.571002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 414/814] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:15.571002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedd4ab0-ae6e-497c-9dcb-c6e533d61222 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-00c40ecb-b621-4b05-a72d-6009f382d216 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:15.584686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:15.585802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:15.586858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:15.586858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:15.586858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:15.586858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:15.588446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 401/815] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:15.589385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c40ecb-b621-4b05-a72d-6009f382d216 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:15.678497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedd4ab0-ae6e-497c-9dcb-c6e533d61222 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:15.705189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c40ecb-b621-4b05-a72d-6009f382d216 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:15.728250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191dcb37-662c-4342-a215-e3c4b64c1995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:15.729162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191dcb37-662c-4342-a215-e3c4b64c1995 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:15.729548 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191dcb37-662c-4342-a215-e3c4b64c1995 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:15.729989 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191dcb37-662c-4342-a215-e3c4b64c1995 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:15.730796 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191dcb37-662c-4342-a215-e3c4b64c1995 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:15.732189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191dcb37-662c-4342-a215-e3c4b64c1995 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:15.741460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a88c91-3421-4033-a7a9-bb2f7c31e740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:15.742133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a88c91-3421-4033-a7a9-bb2f7c31e740 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:15.742407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a88c91-3421-4033-a7a9-bb2f7c31e740 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:15.742676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a88c91-3421-4033-a7a9-bb2f7c31e740 None admin] PATH_INFO: `/v3/projects/8b41573c9f3146d19a306089d686015c/users/c11f1dd5a6b042c9b864a5861d0f3569/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:15.744984 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a88c91-3421-4033-a7a9-bb2f7c31e740 None admin] RBAC: Authorizing `identity:create_grant(project_id=8b41573c9f3146d19a306089d686015c, user_id=c11f1dd5a6b042c9b864a5861d0f3569, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:15.746571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a88c91-3421-4033-a7a9-bb2f7c31e740 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:15.763385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 415/816] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 00:55:15 2022] POST /identity/v3/endpoints => generated 346 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:15.763385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158bb19f-9ce0-4a63-bcee-ec198cad6163 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:15.763385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158bb19f-9ce0-4a63-bcee-ec198cad6163 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:15.763385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158bb19f-9ce0-4a63-bcee-ec198cad6163 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:15.763385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 416/817] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:15.763385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158bb19f-9ce0-4a63-bcee-ec198cad6163 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:15.767056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:15.771877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 402/818] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:55:15 2022] PUT /identity/v3/projects/8b41573c9f3146d19a306089d686015c/users/c11f1dd5a6b042c9b864a5861d0f3569/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:55:15.773448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8556010-b0f0-4b9b-b1d4-5368e8e4f5a1 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:15.773813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8556010-b0f0-4b9b-b1d4-5368e8e4f5a1 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:15.774139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8556010-b0f0-4b9b-b1d4-5368e8e4f5a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:15.798426 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c8556010-b0f0-4b9b-b1d4-5368e8e4f5a1 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:15.799958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c8556010-b0f0-4b9b-b1d4-5368e8e4f5a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:15.874913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:15.913156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 403/819] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:15 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:15.966770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8556010-b0f0-4b9b-b1d4-5368e8e4f5a1 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:15.982482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158bb19f-9ce0-4a63-bcee-ec198cad6163 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:16.038186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9672ee27-19c2-4016-8fd3-bd247e3cd16b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:16.039147 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9672ee27-19c2-4016-8fd3-bd247e3cd16b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:16.039518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9672ee27-19c2-4016-8fd3-bd247e3cd16b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:16.039821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9672ee27-19c2-4016-8fd3-bd247e3cd16b None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:16.040859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9672ee27-19c2-4016-8fd3-bd247e3cd16b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:16.041162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9672ee27-19c2-4016-8fd3-bd247e3cd16b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:16.042822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9672ee27-19c2-4016-8fd3-bd247e3cd16b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:16.061571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 417/820] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:16.067879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9672ee27-19c2-4016-8fd3-bd247e3cd16b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:16.107843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8556010-b0f0-4b9b-b1d4-5368e8e4f5a1 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:16.148634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd385877-7335-4cfc-b246-e45f4d7aebfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:16.149479 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd385877-7335-4cfc-b246-e45f4d7aebfd None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:16.149857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd385877-7335-4cfc-b246-e45f4d7aebfd None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:16.150231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd385877-7335-4cfc-b246-e45f4d7aebfd None admin] PATH_INFO: `/v3/services/network` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:16.151057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd385877-7335-4cfc-b246-e45f4d7aebfd None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:16.152658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd385877-7335-4cfc-b246-e45f4d7aebfd None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:16.154953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-bd385877-7335-4cfc-b246-e45f4d7aebfd None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:16.164515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:16.165638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 31 00:55:16.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:16.169447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 404/821] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:15 2022] GET /identity/v3/services/network => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:16.176407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd385877-7335-4cfc-b246-e45f4d7aebfd None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:16.208106 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9672ee27-19c2-4016-8fd3-bd247e3cd16b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:16.232599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:16.272204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ac7589-ea87-4deb-ae26-b4f06276dcf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:16.272713 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ac7589-ea87-4deb-ae26-b4f06276dcf3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:16.272992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ac7589-ea87-4deb-ae26-b4f06276dcf3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:16.273259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ac7589-ea87-4deb-ae26-b4f06276dcf3 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:16.301079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ac7589-ea87-4deb-ae26-b4f06276dcf3 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:16.302380 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ac7589-ea87-4deb-ae26-b4f06276dcf3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:16.310790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd385877-7335-4cfc-b246-e45f4d7aebfd None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-71ac7589-ea87-4deb-ae26-b4f06276dcf3 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:16.329632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:16.330835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:16.332020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:16.332020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:16.332020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:16.332020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:16.332020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:16.332020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:16.332020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:16.332020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:16.332020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:55:16.332020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:16.339841 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 418/822] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Jul 31 00:55:16 2022] GET /identity/v3/users/nova => generated 82 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:16.345420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ac7589-ea87-4deb-ae26-b4f06276dcf3 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:16.356047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2f602e-407d-4746-8f74-6818800dd39e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:16.357029 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2f602e-407d-4746-8f74-6818800dd39e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:16.359786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2f602e-407d-4746-8f74-6818800dd39e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:16.360073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2f602e-407d-4746-8f74-6818800dd39e None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:16.360776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2f602e-407d-4746-8f74-6818800dd39e None admin] RBAC: Adding query filter params (name=network) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:16.361080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2f602e-407d-4746-8f74-6818800dd39e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:16.362474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2f602e-407d-4746-8f74-6818800dd39e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:16.376113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 405/823] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:16 2022] GET /identity/v3/services?name=network => generated 124 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:16.381511 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2f602e-407d-4746-8f74-6818800dd39e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:16.467821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ac7589-ea87-4deb-ae26-b4f06276dcf3 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:16.505967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:16.537285 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de96804-c1e0-4297-9b49-97f73c8b4188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:16.537916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de96804-c1e0-4297-9b49-97f73c8b4188 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:16.538193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de96804-c1e0-4297-9b49-97f73c8b4188 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:16.538462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de96804-c1e0-4297-9b49-97f73c8b4188 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:16.539387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de96804-c1e0-4297-9b49-97f73c8b4188 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:16.539799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de96804-c1e0-4297-9b49-97f73c8b4188 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:16.541559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de96804-c1e0-4297-9b49-97f73c8b4188 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:16.557505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2f602e-407d-4746-8f74-6818800dd39e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:16.569407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:16.584144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 419/824] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Jul 31 00:55:16 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:16.586735 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c508a3ba-2e5a-43b3-b292-db382eeaa456 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:16.587016 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c508a3ba-2e5a-43b3-b292-db382eeaa456 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:16.587294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c508a3ba-2e5a-43b3-b292-db382eeaa456 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:16.592872 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 420/825] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:55:16.592872 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c508a3ba-2e5a-43b3-b292-db382eeaa456 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:16.600907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d179567f-bee4-454c-96bf-a11d9a92813b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:16.601916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179567f-bee4-454c-96bf-a11d9a92813b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:16.602402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179567f-bee4-454c-96bf-a11d9a92813b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:16.602899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179567f-bee4-454c-96bf-a11d9a92813b None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:16.603771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d179567f-bee4-454c-96bf-a11d9a92813b None admin] RBAC: Adding query filter params (type=network) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:16.604244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d179567f-bee4-454c-96bf-a11d9a92813b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:16.605760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d179567f-bee4-454c-96bf-a11d9a92813b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:16.611793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 406/826] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:16 2022] GET /identity/v3/services?type=network => generated 350 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:16.613240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54feb0e1-3f5b-45b8-bdb2-eca41c7cec0d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:16.613743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54feb0e1-3f5b-45b8-bdb2-eca41c7cec0d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:16.614245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54feb0e1-3f5b-45b8-bdb2-eca41c7cec0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:16.633123 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-54feb0e1-3f5b-45b8-bdb2-eca41c7cec0d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:16.634684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-54feb0e1-3f5b-45b8-bdb2-eca41c7cec0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:16.715380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 407/827] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:16.715380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54feb0e1-3f5b-45b8-bdb2-eca41c7cec0d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:16.725241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c508a3ba-2e5a-43b3-b292-db382eeaa456 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:16.768824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fb6586-0836-40db-a68a-2b6e2ef8c93a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:16.769431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fb6586-0836-40db-a68a-2b6e2ef8c93a None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:16.769707 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fb6586-0836-40db-a68a-2b6e2ef8c93a None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:16.769978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fb6586-0836-40db-a68a-2b6e2ef8c93a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:16.824755 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:16.836646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54feb0e1-3f5b-45b8-bdb2-eca41c7cec0d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:16.839797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:16.847599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fb6586-0836-40db-a68a-2b6e2ef8c93a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:16.849365 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fb6586-0836-40db-a68a-2b6e2ef8c93a None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:16.876115 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f54235-06b7-4104-a554-47586d5861b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:16.877012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f54235-06b7-4104-a554-47586d5861b7 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:16.877303 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f54235-06b7-4104-a554-47586d5861b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:16.877581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f54235-06b7-4104-a554-47586d5861b7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:16.878296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f54235-06b7-4104-a554-47586d5861b7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:16.880206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f54235-06b7-4104-a554-47586d5861b7 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:16.909596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 408/828] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 00:55:16 2022] POST /identity/v3/endpoints => generated 327 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:16.911759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f54235-06b7-4104-a554-47586d5861b7 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:16.969272 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 421/829] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:16 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:55:16.975190 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fb6586-0836-40db-a68a-2b6e2ef8c93a None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:17.088553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f54235-06b7-4104-a554-47586d5861b7 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:17.127428 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:17.165750 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fb6586-0836-40db-a68a-2b6e2ef8c93a None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:17.168068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1817aa-1661-4ec1-9911-f22d381c4e16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:17.169654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1817aa-1661-4ec1-9911-f22d381c4e16 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:17.169654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1817aa-1661-4ec1-9911-f22d381c4e16 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:17.169654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1817aa-1661-4ec1-9911-f22d381c4e16 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:17.186875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1817aa-1661-4ec1-9911-f22d381c4e16 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-fa1817aa-1661-4ec1-9911-f22d381c4e16 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:17.193821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:17.194850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:17.195736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 409/830] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:55:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:17.205296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1817aa-1661-4ec1-9911-f22d381c4e16 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:17.227948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b016bf-218c-48d7-bfe3-36a068c4f646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:17.228890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b016bf-218c-48d7-bfe3-36a068c4f646 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:17.229324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b016bf-218c-48d7-bfe3-36a068c4f646 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:17.229764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b016bf-218c-48d7-bfe3-36a068c4f646 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:17.231399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:17.250010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b016bf-218c-48d7-bfe3-36a068c4f646 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:17.254012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b016bf-218c-48d7-bfe3-36a068c4f646 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-56b016bf-218c-48d7-bfe3-36a068c4f646 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:17.275547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:17.277734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:17.279878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:17.279878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:17.279878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:17.279878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:17.279878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 422/831] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:17.283052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b016bf-218c-48d7-bfe3-36a068c4f646 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:17.367080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1817aa-1661-4ec1-9911-f22d381c4e16 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:17.400271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b016bf-218c-48d7-bfe3-36a068c4f646 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:17.412614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa261adc-3383-406d-9584-2d2ba470ba38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:17.413663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa261adc-3383-406d-9584-2d2ba470ba38 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:17.414247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa261adc-3383-406d-9584-2d2ba470ba38 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:17.414754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa261adc-3383-406d-9584-2d2ba470ba38 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:17.415762 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa261adc-3383-406d-9584-2d2ba470ba38 None admin] RBAC: Adding query filter params (name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:17.416283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa261adc-3383-406d-9584-2d2ba470ba38 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:17.417983 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa261adc-3383-406d-9584-2d2ba470ba38 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:17.433195 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 410/832] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:55:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:17.433718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:17.435551 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa261adc-3383-406d-9584-2d2ba470ba38 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:17.437698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d136987a-2f32-414d-a506-cfe4c7c814cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:17.438466 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d136987a-2f32-414d-a506-cfe4c7c814cb None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:17.438857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d136987a-2f32-414d-a506-cfe4c7c814cb None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:17.439240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d136987a-2f32-414d-a506-cfe4c7c814cb None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:17.440074 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d136987a-2f32-414d-a506-cfe4c7c814cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:17.440483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d136987a-2f32-414d-a506-cfe4c7c814cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:17.442256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d136987a-2f32-414d-a506-cfe4c7c814cb None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:17.456071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 423/833] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:17.461170 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d136987a-2f32-414d-a506-cfe4c7c814cb None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:17.524644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:17.553825 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa261adc-3383-406d-9584-2d2ba470ba38 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:17.576370 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d136987a-2f32-414d-a506-cfe4c7c814cb None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:17.595555 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f74f5b-f118-4179-98a5-301f95f8967c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:17.596553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f74f5b-f118-4179-98a5-301f95f8967c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:17.597110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f74f5b-f118-4179-98a5-301f95f8967c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:17.597564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f74f5b-f118-4179-98a5-301f95f8967c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:17.611181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f74f5b-f118-4179-98a5-301f95f8967c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:17.613109 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f74f5b-f118-4179-98a5-301f95f8967c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:17.617444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bb6ae0-1dee-434b-9495-50f8d9b5c71b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:17.618123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bb6ae0-1dee-434b-9495-50f8d9b5c71b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:17.618471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bb6ae0-1dee-434b-9495-50f8d9b5c71b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:17.618825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bb6ae0-1dee-434b-9495-50f8d9b5c71b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:17.626930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bb6ae0-1dee-434b-9495-50f8d9b5c71b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:17.628616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bb6ae0-1dee-434b-9495-50f8d9b5c71b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-b4f74f5b-f118-4179-98a5-301f95f8967c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:17.632122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:17.633023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:17.633919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:17.633919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:17.633919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:17.633919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:17.641491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 411/834] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-64bb6ae0-1dee-434b-9495-50f8d9b5c71b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:17.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:17.643640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:17.644839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:17.644839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:17.644839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:17.644839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:17.644839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:17.644839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:17.644839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:17.644839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:17.645972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f74f5b-f118-4179-98a5-301f95f8967c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:17.648888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 424/835] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:17.649980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bb6ae0-1dee-434b-9495-50f8d9b5c71b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:17.705974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:17.755975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bb6ae0-1dee-434b-9495-50f8d9b5c71b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:17.763297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f74f5b-f118-4179-98a5-301f95f8967c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:17.792443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7822945-d482-4ebc-86a1-228b45c30038 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:17.793067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7822945-d482-4ebc-86a1-228b45c30038 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:17.793360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7822945-d482-4ebc-86a1-228b45c30038 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:17.793631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7822945-d482-4ebc-86a1-228b45c30038 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:17.794389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7822945-d482-4ebc-86a1-228b45c30038 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:17.794706 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7822945-d482-4ebc-86a1-228b45c30038 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:17.796202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7822945-d482-4ebc-86a1-228b45c30038 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:17.798031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:17.806291 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16423de0-805b-414c-a9cb-c2bdba9a2dc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:17.807251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16423de0-805b-414c-a9cb-c2bdba9a2dc5 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:17.807765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16423de0-805b-414c-a9cb-c2bdba9a2dc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:17.808238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16423de0-805b-414c-a9cb-c2bdba9a2dc5 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:17.809126 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16423de0-805b-414c-a9cb-c2bdba9a2dc5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:17.809653 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16423de0-805b-414c-a9cb-c2bdba9a2dc5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:17.811412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16423de0-805b-414c-a9cb-c2bdba9a2dc5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:17.814670 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 425/836] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:17.815934 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7822945-d482-4ebc-86a1-228b45c30038 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:17.830402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 412/837] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:17.835655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16423de0-805b-414c-a9cb-c2bdba9a2dc5 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:17.934047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7822945-d482-4ebc-86a1-228b45c30038 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:17.949032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:17.960465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16423de0-805b-414c-a9cb-c2bdba9a2dc5 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:17.972627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcffecf5-f9d6-4de1-b533-9f5c2eed822d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:17.973254 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcffecf5-f9d6-4de1-b533-9f5c2eed822d None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:17.973534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcffecf5-f9d6-4de1-b533-9f5c2eed822d None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:17.973809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcffecf5-f9d6-4de1-b533-9f5c2eed822d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:17.974607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcffecf5-f9d6-4de1-b533-9f5c2eed822d None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=5b0bd572ae7a4c7f9e07fe0b9708aa7e) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:17.975188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcffecf5-f9d6-4de1-b533-9f5c2eed822d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:17.976526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcffecf5-f9d6-4de1-b533-9f5c2eed822d None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:17.987655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 426/838] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:17 2022] GET /identity/v3/role_assignments?user.id=5b0bd572ae7a4c7f9e07fe0b9708aa7e&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:18.005563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e060752-6910-49c5-8c38-f9e5e1ae51b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:18.006716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e060752-6910-49c5-8c38-f9e5e1ae51b1 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:18.007210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e060752-6910-49c5-8c38-f9e5e1ae51b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:18.007817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e060752-6910-49c5-8c38-f9e5e1ae51b1 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:18.018149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e060752-6910-49c5-8c38-f9e5e1ae51b1 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:18.020432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e060752-6910-49c5-8c38-f9e5e1ae51b1 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-6e060752-6910-49c5-8c38-f9e5e1ae51b1 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:18.035637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:18.036664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:18.037569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:18.037569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:18.037569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:18.037569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:18.037569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:18.037569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:18.037569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:18.037569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:18.037569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 31 00:55:18.037569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:18.045149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 413/839] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Jul 31 00:55:17 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:18.047660 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcffecf5-f9d6-4de1-b533-9f5c2eed822d None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:18.167071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcffecf5-f9d6-4de1-b533-9f5c2eed822d None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:18.170728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4917e89-c4e2-4bb5-890d-e5e8c4c498a6 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:18.171019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4917e89-c4e2-4bb5-890d-e5e8c4c498a6 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:18.171332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4917e89-c4e2-4bb5-890d-e5e8c4c498a6 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:18.172946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 414/840] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:18.178262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e53f0c1-8773-4946-ba95-29ac9a7d038f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:18.178542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e53f0c1-8773-4946-ba95-29ac9a7d038f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:18.178814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e53f0c1-8773-4946-ba95-29ac9a7d038f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:18.203963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6e53f0c1-8773-4946-ba95-29ac9a7d038f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:18.205107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6e53f0c1-8773-4946-ba95-29ac9a7d038f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:18.205965 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba10158-4d29-496e-a434-24680a204edd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:18.206692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba10158-4d29-496e-a434-24680a204edd None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:18.207188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba10158-4d29-496e-a434-24680a204edd None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:18.207432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba10158-4d29-496e-a434-24680a204edd None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:18.208171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba10158-4d29-496e-a434-24680a204edd None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:18.208475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba10158-4d29-496e-a434-24680a204edd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:18.209919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba10158-4d29-496e-a434-24680a204edd None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:18.229326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 427/841] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Jul 31 00:55:18 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:18.230436 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efcfa81-e9f1-4461-b6e3-f8620c7b4757 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:18.230717 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efcfa81-e9f1-4461-b6e3-f8620c7b4757 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:18.230989 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efcfa81-e9f1-4461-b6e3-f8620c7b4757 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:18.232320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 428/842] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:18.235710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efcfa81-e9f1-4461-b6e3-f8620c7b4757 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:18.270156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:18.276584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:18.295915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 415/843] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:18 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:18.297814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a652b4a-14db-48e4-b6d7-58fba00feff2 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:18.298153 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a652b4a-14db-48e4-b6d7-58fba00feff2 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:18.298486 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a652b4a-14db-48e4-b6d7-58fba00feff2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:18.320702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-7a652b4a-14db-48e4-b6d7-58fba00feff2 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:18.322162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7a652b4a-14db-48e4-b6d7-58fba00feff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:18.382576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efcfa81-e9f1-4461-b6e3-f8620c7b4757 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:18.403913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 416/844] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:18 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:18.404904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a652b4a-14db-48e4-b6d7-58fba00feff2 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:18.428835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7602bafc-cae2-4eb5-a6c9-6148b267cc9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:18.430173 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7602bafc-cae2-4eb5-a6c9-6148b267cc9f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:18.430741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7602bafc-cae2-4eb5-a6c9-6148b267cc9f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:18.431342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7602bafc-cae2-4eb5-a6c9-6148b267cc9f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:18.508341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7602bafc-cae2-4eb5-a6c9-6148b267cc9f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:18.510065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7602bafc-cae2-4eb5-a6c9-6148b267cc9f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:18.528334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a652b4a-14db-48e4-b6d7-58fba00feff2 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:18.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:18.558654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:18.575730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d825cd-c9a8-44f9-97fe-1236027e76c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:18.576676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d825cd-c9a8-44f9-97fe-1236027e76c0 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:18.577163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d825cd-c9a8-44f9-97fe-1236027e76c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:18.577646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d825cd-c9a8-44f9-97fe-1236027e76c0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:18.591964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d825cd-c9a8-44f9-97fe-1236027e76c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-c8d825cd-c9a8-44f9-97fe-1236027e76c0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:18.600840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:18.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:18.603272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:18.603272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:18.603272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:18.603272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:18.603272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:18.603272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:18.603272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:18.603272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:18.603272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:18.603272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:18.603272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:18.603272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:18.613921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 417/845] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:18.615840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d825cd-c9a8-44f9-97fe-1236027e76c0 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:18.645432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 429/846] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:18 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:55:18.646953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7602bafc-cae2-4eb5-a6c9-6148b267cc9f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:18.774015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d825cd-c9a8-44f9-97fe-1236027e76c0 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:18.823112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7602bafc-cae2-4eb5-a6c9-6148b267cc9f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:18.836092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d026fb-455a-40ca-812f-10f2df7d464e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:18.837358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d026fb-455a-40ca-812f-10f2df7d464e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:18.837913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d026fb-455a-40ca-812f-10f2df7d464e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:18.838379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d026fb-455a-40ca-812f-10f2df7d464e None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:18.839751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d026fb-455a-40ca-812f-10f2df7d464e None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:18.842724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d026fb-455a-40ca-812f-10f2df7d464e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:18.860636 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95fb47c-d20f-4fb0-aa56-2a0eb9ab95c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:18.861248 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95fb47c-d20f-4fb0-aa56-2a0eb9ab95c4 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:18.861523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95fb47c-d20f-4fb0-aa56-2a0eb9ab95c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:18.861791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95fb47c-d20f-4fb0-aa56-2a0eb9ab95c4 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:18.862497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95fb47c-d20f-4fb0-aa56-2a0eb9ab95c4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:18.862797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95fb47c-d20f-4fb0-aa56-2a0eb9ab95c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-16d026fb-455a-40ca-812f-10f2df7d464e None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:18.863006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:18.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 31 00:55:18.868900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:18.873538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95fb47c-d20f-4fb0-aa56-2a0eb9ab95c4 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:18.873538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:18.873944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 418/847] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:55:18 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:18.873944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d026fb-455a-40ca-812f-10f2df7d464e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:18.879894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 430/848] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:55:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:18.885069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95fb47c-d20f-4fb0-aa56-2a0eb9ab95c4 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:18.918813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:18.990893 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d026fb-455a-40ca-812f-10f2df7d464e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:18.991596 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95fb47c-d20f-4fb0-aa56-2a0eb9ab95c4 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:19.025835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c4a419-e2d9-4e98-8e2f-310655b1b020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:19.026010 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038b74f0-0b70-4a84-87c6-2dfa1295ad8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:19.026368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c4a419-e2d9-4e98-8e2f-310655b1b020 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:19.026471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b74f0-0b70-4a84-87c6-2dfa1295ad8b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:19.026655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c4a419-e2d9-4e98-8e2f-310655b1b020 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:19.026749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b74f0-0b70-4a84-87c6-2dfa1295ad8b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:19.026905 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c4a419-e2d9-4e98-8e2f-310655b1b020 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:19.026998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b74f0-0b70-4a84-87c6-2dfa1295ad8b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:19.027692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c4a419-e2d9-4e98-8e2f-310655b1b020 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:19.027994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c4a419-e2d9-4e98-8e2f-310655b1b020 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:19.029358 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c4a419-e2d9-4e98-8e2f-310655b1b020 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:19.039134 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 431/849] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:55:18 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:19.039134 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c4a419-e2d9-4e98-8e2f-310655b1b020 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:19.044968 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038b74f0-0b70-4a84-87c6-2dfa1295ad8b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:19.046616 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038b74f0-0b70-4a84-87c6-2dfa1295ad8b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-038b74f0-0b70-4a84-87c6-2dfa1295ad8b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:19.064777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:19.065729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:19.066613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:19.066613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:19.066613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:19.066613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:19.066613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 419/850] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:19.066899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038b74f0-0b70-4a84-87c6-2dfa1295ad8b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:19.112948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:19.160456 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c4a419-e2d9-4e98-8e2f-310655b1b020 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:19.175108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:19.195516 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038b74f0-0b70-4a84-87c6-2dfa1295ad8b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:19.198373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501526ad-0459-4d07-b234-f9a1db7795c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:19.199046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501526ad-0459-4d07-b234-f9a1db7795c9 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:19.199332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501526ad-0459-4d07-b234-f9a1db7795c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:19.199640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501526ad-0459-4d07-b234-f9a1db7795c9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:19.213510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501526ad-0459-4d07-b234-f9a1db7795c9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:19.214994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501526ad-0459-4d07-b234-f9a1db7795c9 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-501526ad-0459-4d07-b234-f9a1db7795c9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:19.234542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:19.235639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:19.236512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:19.236512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:19.236512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:19.236512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:19.240471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 432/851] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:19.241357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501526ad-0459-4d07-b234-f9a1db7795c9 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:19.242267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee119f41-5499-45e7-a14d-a084fd387816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:19.242981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee119f41-5499-45e7-a14d-a084fd387816 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:19.243474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee119f41-5499-45e7-a14d-a084fd387816 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:19.243774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee119f41-5499-45e7-a14d-a084fd387816 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:19.245077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee119f41-5499-45e7-a14d-a084fd387816 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:19.245077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee119f41-5499-45e7-a14d-a084fd387816 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:19.246412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee119f41-5499-45e7-a14d-a084fd387816 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:19.251831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 420/852] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:55:19 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:19.252517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee119f41-5499-45e7-a14d-a084fd387816 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:19.385103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501526ad-0459-4d07-b234-f9a1db7795c9 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:19.402986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:19.406718 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee119f41-5499-45e7-a14d-a084fd387816 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:19.433866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c95ee0-89b3-430e-89d4-0da8d60f2298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:19.434873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c95ee0-89b3-430e-89d4-0da8d60f2298 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:19.434873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c95ee0-89b3-430e-89d4-0da8d60f2298 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:19.435201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c95ee0-89b3-430e-89d4-0da8d60f2298 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:19.435992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c95ee0-89b3-430e-89d4-0da8d60f2298 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:19.436346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c95ee0-89b3-430e-89d4-0da8d60f2298 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:19.437752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c95ee0-89b3-430e-89d4-0da8d60f2298 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:19.453959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 433/853] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:19.457335 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c0849-4730-4e87-9d67-d7ea2f1a6897 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:19.458154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c0849-4730-4e87-9d67-d7ea2f1a6897 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:19.458261 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c0849-4730-4e87-9d67-d7ea2f1a6897 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:19.458491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c0849-4730-4e87-9d67-d7ea2f1a6897 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:19.459191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39c0849-4730-4e87-9d67-d7ea2f1a6897 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:19.460397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c95ee0-89b3-430e-89d4-0da8d60f2298 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:19.460933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39c0849-4730-4e87-9d67-d7ea2f1a6897 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:19.462335 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39c0849-4730-4e87-9d67-d7ea2f1a6897 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:19.482599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 421/854] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:19.489982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c0849-4730-4e87-9d67-d7ea2f1a6897 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:19.504665 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:19.594879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c95ee0-89b3-430e-89d4-0da8d60f2298 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:19.627700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c0849-4730-4e87-9d67-d7ea2f1a6897 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:19.651508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467cecfa-0c2b-4885-a8a5-f1e6bbe2c880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:19.652231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467cecfa-0c2b-4885-a8a5-f1e6bbe2c880 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:19.652510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467cecfa-0c2b-4885-a8a5-f1e6bbe2c880 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:19.652784 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467cecfa-0c2b-4885-a8a5-f1e6bbe2c880 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:19.667442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467cecfa-0c2b-4885-a8a5-f1e6bbe2c880 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:19.670360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467cecfa-0c2b-4885-a8a5-f1e6bbe2c880 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:19.674175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dafab5-01d7-4d66-b882-7098c424b631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:19.675155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dafab5-01d7-4d66-b882-7098c424b631 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:19.675649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dafab5-01d7-4d66-b882-7098c424b631 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:19.676366 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dafab5-01d7-4d66-b882-7098c424b631 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:19.684972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4dafab5-01d7-4d66-b882-7098c424b631 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:19.686757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4dafab5-01d7-4d66-b882-7098c424b631 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-467cecfa-0c2b-4885-a8a5-f1e6bbe2c880 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:19.688256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:19.689139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:19.690023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:19.690023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:19.690023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:19.690023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:19.690023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:19.690023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:19.690023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:19.690023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:19.695423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 434/855] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:19.701032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467cecfa-0c2b-4885-a8a5-f1e6bbe2c880 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-f4dafab5-01d7-4d66-b882-7098c424b631 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:19.707970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:19.708938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:19.709862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:19.709862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:19.709862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:19.709862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:19.709862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:19.709862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:19.709862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:19.709862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:19.709862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 31 00:55:19.709862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:19.712556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 422/856] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:19 2022] GET /identity/v3/users/swift => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:19.723378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dafab5-01d7-4d66-b882-7098c424b631 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:19.754284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:19.819229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:19.858491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dafab5-01d7-4d66-b882-7098c424b631 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:19.863612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467cecfa-0c2b-4885-a8a5-f1e6bbe2c880 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:19.898531 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b29807e-abd1-4280-a0b7-6524f32c5e66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:19.899455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b29807e-abd1-4280-a0b7-6524f32c5e66 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:19.899747 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b29807e-abd1-4280-a0b7-6524f32c5e66 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:19.900072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b29807e-abd1-4280-a0b7-6524f32c5e66 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:19.900946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b29807e-abd1-4280-a0b7-6524f32c5e66 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:19.901319 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b29807e-abd1-4280-a0b7-6524f32c5e66 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:19.902891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b29807e-abd1-4280-a0b7-6524f32c5e66 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:19.917935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815cf5b8-d7e0-4d53-a52c-42f8374b8e5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:19.918801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815cf5b8-d7e0-4d53-a52c-42f8374b8e5b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:19.919909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815cf5b8-d7e0-4d53-a52c-42f8374b8e5b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:19.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815cf5b8-d7e0-4d53-a52c-42f8374b8e5b None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:19.921585 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815cf5b8-d7e0-4d53-a52c-42f8374b8e5b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:19.922032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815cf5b8-d7e0-4d53-a52c-42f8374b8e5b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:19.928353 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 423/857] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Jul 31 00:55:19 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:19.928511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815cf5b8-d7e0-4d53-a52c-42f8374b8e5b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:19.931159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b29807e-abd1-4280-a0b7-6524f32c5e66 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:19.945317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 435/858] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:19.951019 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815cf5b8-d7e0-4d53-a52c-42f8374b8e5b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:20.052922 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b29807e-abd1-4280-a0b7-6524f32c5e66 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:20.083829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:20.085190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:20.094101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09a7708-8cbe-48cd-b50c-668897e85c33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:20.094718 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09a7708-8cbe-48cd-b50c-668897e85c33 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:20.095002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09a7708-8cbe-48cd-b50c-668897e85c33 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:20.095274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09a7708-8cbe-48cd-b50c-668897e85c33 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:20.099031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815cf5b8-d7e0-4d53-a52c-42f8374b8e5b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:20.138452 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc9f6d6-4471-4d00-be3b-6f74ee21c993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:20.139552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc9f6d6-4471-4d00-be3b-6f74ee21c993 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:20.140634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc9f6d6-4471-4d00-be3b-6f74ee21c993 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:20.140913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc9f6d6-4471-4d00-be3b-6f74ee21c993 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:20.141716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc9f6d6-4471-4d00-be3b-6f74ee21c993 None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=8cdffe91785e42d798b931082969598e) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:20.142024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc9f6d6-4471-4d00-be3b-6f74ee21c993 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:20.143534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc9f6d6-4471-4d00-be3b-6f74ee21c993 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:20.155999 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 436/859] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:19 2022] GET /identity/v3/role_assignments?user.id=8cdffe91785e42d798b931082969598e&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 608 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:20.158526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c178b65-af0f-4c98-a71b-5d87a104105e None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:20.159247 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c178b65-af0f-4c98-a71b-5d87a104105e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:20.159694 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c178b65-af0f-4c98-a71b-5d87a104105e None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:20.161654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 437/860] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:20 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:20.167647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933bb837-b1df-4e40-98f8-2207f682f4a9 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:20.168333 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933bb837-b1df-4e40-98f8-2207f682f4a9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:20.168333 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933bb837-b1df-4e40-98f8-2207f682f4a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:20.176932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09a7708-8cbe-48cd-b50c-668897e85c33 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:20.178799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09a7708-8cbe-48cd-b50c-668897e85c33 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:20.188513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-933bb837-b1df-4e40-98f8-2207f682f4a9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:20.189850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-933bb837-b1df-4e40-98f8-2207f682f4a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:20.272136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 438/861] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:20 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:20.287700 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933bb837-b1df-4e40-98f8-2207f682f4a9 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:20.303208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 424/862] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:19 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:55:20.308797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09a7708-8cbe-48cd-b50c-668897e85c33 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:20.347444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:20.400315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:20.422483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933bb837-b1df-4e40-98f8-2207f682f4a9 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:20.463376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d51cf0-3227-4ec1-acf2-f6b8984d8519 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:20.464137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d51cf0-3227-4ec1-acf2-f6b8984d8519 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:20.464354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d51cf0-3227-4ec1-acf2-f6b8984d8519 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:20.464598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d51cf0-3227-4ec1-acf2-f6b8984d8519 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:20.465250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d51cf0-3227-4ec1-acf2-f6b8984d8519 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:20.465751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09a7708-8cbe-48cd-b50c-668897e85c33 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:20.466673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d51cf0-3227-4ec1-acf2-f6b8984d8519 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-19d51cf0-3227-4ec1-acf2-f6b8984d8519 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:20.477500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:20.478514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:20.479838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 439/863] 10.222.0.42 () {56 vars in 1132 bytes} [Sun Jul 31 00:55:20 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:20.486602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d51cf0-3227-4ec1-acf2-f6b8984d8519 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:20.515884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6364ae-a6eb-4a56-a0c7-6f3674ff6c2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:20.517370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6364ae-a6eb-4a56-a0c7-6f3674ff6c2e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:20.518109 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6364ae-a6eb-4a56-a0c7-6f3674ff6c2e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:20.518599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6364ae-a6eb-4a56-a0c7-6f3674ff6c2e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:20.531722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6364ae-a6eb-4a56-a0c7-6f3674ff6c2e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:20.533923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6364ae-a6eb-4a56-a0c7-6f3674ff6c2e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-bd6364ae-a6eb-4a56-a0c7-6f3674ff6c2e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:20.554451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:20.555326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:20.556431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:20.556431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:20.556431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:20.556431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:20.559087 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 425/864] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:20.564921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6364ae-a6eb-4a56-a0c7-6f3674ff6c2e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:20.633536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d51cf0-3227-4ec1-acf2-f6b8984d8519 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:20.676440 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1792157-0155-4be8-890a-3e80d386d56f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:20.677407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1792157-0155-4be8-890a-3e80d386d56f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:20.677836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1792157-0155-4be8-890a-3e80d386d56f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:20.678623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:20.679204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1792157-0155-4be8-890a-3e80d386d56f None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:20.680483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1792157-0155-4be8-890a-3e80d386d56f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:20.680960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1792157-0155-4be8-890a-3e80d386d56f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:20.682904 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1792157-0155-4be8-890a-3e80d386d56f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:20.690675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 440/865] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Jul 31 00:55:20 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:20.696508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1792157-0155-4be8-890a-3e80d386d56f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:20.712345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6364ae-a6eb-4a56-a0c7-6f3674ff6c2e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:20.723716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:20.750788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc0a245-6c7a-4a3b-aa59-3967a17281b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:20.752197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc0a245-6c7a-4a3b-aa59-3967a17281b1 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:20.752776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc0a245-6c7a-4a3b-aa59-3967a17281b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:20.753245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc0a245-6c7a-4a3b-aa59-3967a17281b1 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:20.754160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc0a245-6c7a-4a3b-aa59-3967a17281b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:20.754659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc0a245-6c7a-4a3b-aa59-3967a17281b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:20.756252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc0a245-6c7a-4a3b-aa59-3967a17281b1 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:20.769475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 426/866] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:20.774668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc0a245-6c7a-4a3b-aa59-3967a17281b1 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:20.813640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1792157-0155-4be8-890a-3e80d386d56f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:20.850881 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bc83d7-4b13-413f-92fc-85f19fcf31e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:20.851763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bc83d7-4b13-413f-92fc-85f19fcf31e4 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:20.852056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bc83d7-4b13-413f-92fc-85f19fcf31e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:20.852329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bc83d7-4b13-413f-92fc-85f19fcf31e4 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:20.853048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35bc83d7-4b13-413f-92fc-85f19fcf31e4 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:20.853346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35bc83d7-4b13-413f-92fc-85f19fcf31e4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:20.854701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35bc83d7-4b13-413f-92fc-85f19fcf31e4 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:20.864865 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 441/867] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Jul 31 00:55:20 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:20.913757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc0a245-6c7a-4a3b-aa59-3967a17281b1 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:20.981964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50829b8d-5b09-4209-9ac4-119c431df1b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:20.983047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50829b8d-5b09-4209-9ac4-119c431df1b1 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:20.983658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50829b8d-5b09-4209-9ac4-119c431df1b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:20.984627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50829b8d-5b09-4209-9ac4-119c431df1b1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:20.999841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50829b8d-5b09-4209-9ac4-119c431df1b1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:21.002793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50829b8d-5b09-4209-9ac4-119c431df1b1 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:21.007566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-50829b8d-5b09-4209-9ac4-119c431df1b1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:21.028100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:21.029047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:21.029909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:21.029909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:21.029909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:21.029909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:21.029909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:21.029909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:21.029909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:21.029909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:21.031413 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 427/868] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:21.040752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bc83d7-4b13-413f-92fc-85f19fcf31e4 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:21.187745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:21.225902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bc83d7-4b13-413f-92fc-85f19fcf31e4 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:21.265438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ffec9e-884a-483c-bd43-4ca36820ffc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:21.266156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ffec9e-884a-483c-bd43-4ca36820ffc3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:21.266542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ffec9e-884a-483c-bd43-4ca36820ffc3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:21.266871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ffec9e-884a-483c-bd43-4ca36820ffc3 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:21.267865 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ffec9e-884a-483c-bd43-4ca36820ffc3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:21.268263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ffec9e-884a-483c-bd43-4ca36820ffc3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:21.269776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ffec9e-884a-483c-bd43-4ca36820ffc3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:21.283919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 442/869] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:21.291036 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50829b8d-5b09-4209-9ac4-119c431df1b1 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:21.417209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50829b8d-5b09-4209-9ac4-119c431df1b1 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:21.454878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53017d0-e7ac-43a2-b686-46de302f3b95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:21.455616 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53017d0-e7ac-43a2-b686-46de302f3b95 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:21.455839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53017d0-e7ac-43a2-b686-46de302f3b95 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:21.456113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53017d0-e7ac-43a2-b686-46de302f3b95 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:21.456901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53017d0-e7ac-43a2-b686-46de302f3b95 None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=c11f1dd5a6b042c9b864a5861d0f3569) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:21.457208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53017d0-e7ac-43a2-b686-46de302f3b95 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:21.458846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53017d0-e7ac-43a2-b686-46de302f3b95 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:21.465966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 428/870] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:21 2022] GET /identity/v3/role_assignments?user.id=c11f1dd5a6b042c9b864a5861d0f3569&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:21.550415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7951481-9f3d-4d42-b149-8451ab16f220 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:21.551114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7951481-9f3d-4d42-b149-8451ab16f220 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:21.551291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7951481-9f3d-4d42-b149-8451ab16f220 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:21.553018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 443/871] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:21.558807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1e8849-2a49-416c-ad1e-12d4dc533596 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:21.559137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1e8849-2a49-416c-ad1e-12d4dc533596 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:21.559373 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1e8849-2a49-416c-ad1e-12d4dc533596 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:21.580089 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-fc1e8849-2a49-416c-ad1e-12d4dc533596 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:21.581412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fc1e8849-2a49-416c-ad1e-12d4dc533596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:21.648775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:21.663436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 429/872] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:21 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:21.680836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7951481-9f3d-4d42-b149-8451ab16f220 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:21.798034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7951481-9f3d-4d42-b149-8451ab16f220 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:21.835625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627428eb-a5a6-438d-bc4b-a46f4f39453d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:21.836164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627428eb-a5a6-438d-bc4b-a46f4f39453d None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:21.836440 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627428eb-a5a6-438d-bc4b-a46f4f39453d None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:21.836709 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627428eb-a5a6-438d-bc4b-a46f4f39453d None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:21.837422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627428eb-a5a6-438d-bc4b-a46f4f39453d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:21.838787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627428eb-a5a6-438d-bc4b-a46f4f39453d None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:21.850931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:21.858342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 444/873] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:55:21 2022] POST /identity/v3/services => generated 251 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:22.298974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c54edb-7a35-490d-be9b-07b6e29ddffa None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:22.299627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c54edb-7a35-490d-be9b-07b6e29ddffa None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:22.299627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c54edb-7a35-490d-be9b-07b6e29ddffa None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:22.301418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 430/874] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:22.306795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bf708f-22ce-46c4-9cb2-b6ebd6993394 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:22.307075 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bf708f-22ce-46c4-9cb2-b6ebd6993394 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:22.307381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bf708f-22ce-46c4-9cb2-b6ebd6993394 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:22.327665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f0bf708f-22ce-46c4-9cb2-b6ebd6993394 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:22.329207 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f0bf708f-22ce-46c4-9cb2-b6ebd6993394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:22.409599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 445/875] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:22 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:22.433814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c54edb-7a35-490d-be9b-07b6e29ddffa None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:22.549995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c54edb-7a35-490d-be9b-07b6e29ddffa None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:22.587049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a9685e-f3e9-4e5c-ac77-1d00e4a5312c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:22.587774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a9685e-f3e9-4e5c-ac77-1d00e4a5312c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:22.588050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a9685e-f3e9-4e5c-ac77-1d00e4a5312c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:22.588322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a9685e-f3e9-4e5c-ac77-1d00e4a5312c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:22.599896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a9685e-f3e9-4e5c-ac77-1d00e4a5312c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-74a9685e-f3e9-4e5c-ac77-1d00e4a5312c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:22.604831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:22.606006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:22.607102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 431/876] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Jul 31 00:55:22 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:22.612884 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bf708f-22ce-46c4-9cb2-b6ebd6993394 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:22.721527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bf708f-22ce-46c4-9cb2-b6ebd6993394 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:22.747198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:22.759640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a0ae5b-b430-4995-ad09-db8a347f859d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:22.760269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a0ae5b-b430-4995-ad09-db8a347f859d None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:22.760548 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a0ae5b-b430-4995-ad09-db8a347f859d None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:22.760822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a0ae5b-b430-4995-ad09-db8a347f859d None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:22.761546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a0ae5b-b430-4995-ad09-db8a347f859d None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:22.761936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a0ae5b-b430-4995-ad09-db8a347f859d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:22.763336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a0ae5b-b430-4995-ad09-db8a347f859d None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:22.773324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 446/877] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Jul 31 00:55:22 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:22.778982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a9685e-f3e9-4e5c-ac77-1d00e4a5312c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:22.839774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:22.886675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a9685e-f3e9-4e5c-ac77-1d00e4a5312c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:22.923484 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee893cb-8e1f-4d2a-9d1d-953eb7b4a56b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:22.924120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee893cb-8e1f-4d2a-9d1d-953eb7b4a56b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:22.924394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee893cb-8e1f-4d2a-9d1d-953eb7b4a56b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:22.924660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee893cb-8e1f-4d2a-9d1d-953eb7b4a56b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:22.935767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee893cb-8e1f-4d2a-9d1d-953eb7b4a56b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:22.937250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee893cb-8e1f-4d2a-9d1d-953eb7b4a56b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-eee893cb-8e1f-4d2a-9d1d-953eb7b4a56b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:22.953424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:22.954358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:22.955224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:22.955224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:22.955224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:22.955224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:22.955224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 432/878] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:22.960605 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a0ae5b-b430-4995-ad09-db8a347f859d None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:23.076980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a0ae5b-b430-4995-ad09-db8a347f859d None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:23.101617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57fed91-dabe-4e4f-bd2e-a827cfce39c4 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:23.102322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57fed91-dabe-4e4f-bd2e-a827cfce39c4 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:23.102843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57fed91-dabe-4e4f-bd2e-a827cfce39c4 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:23.105238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 433/879] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:55:23.111778 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6486e5b9-e85f-4386-8acc-7fdac49205fa None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:23.112408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6486e5b9-e85f-4386-8acc-7fdac49205fa None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:23.112929 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6486e5b9-e85f-4386-8acc-7fdac49205fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:23.123400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8df4fbb-4308-4ff2-92a3-a56e0442f3d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:23.124044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8df4fbb-4308-4ff2-92a3-a56e0442f3d2 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:23.124915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8df4fbb-4308-4ff2-92a3-a56e0442f3d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:23.125194 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8df4fbb-4308-4ff2-92a3-a56e0442f3d2 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:23.126016 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8df4fbb-4308-4ff2-92a3-a56e0442f3d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:23.126326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8df4fbb-4308-4ff2-92a3-a56e0442f3d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:23.127816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8df4fbb-4308-4ff2-92a3-a56e0442f3d2 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:23.136160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6486e5b9-e85f-4386-8acc-7fdac49205fa None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:23.137758 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6486e5b9-e85f-4386-8acc-7fdac49205fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:23.142045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 447/880] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:23.152617 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8df4fbb-4308-4ff2-92a3-a56e0442f3d2 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:23.211558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:23.231387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 434/881] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:23 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:23.247118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6486e5b9-e85f-4386-8acc-7fdac49205fa None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:23.287601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8df4fbb-4308-4ff2-92a3-a56e0442f3d2 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:23.289334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:23.337678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6f9166-d9a9-48ba-a523-a268754b06db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:23.339479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6f9166-d9a9-48ba-a523-a268754b06db None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:23.340331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6f9166-d9a9-48ba-a523-a268754b06db None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:23.341142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6f9166-d9a9-48ba-a523-a268754b06db None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:23.355853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6f9166-d9a9-48ba-a523-a268754b06db None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:23.358289 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6f9166-d9a9-48ba-a523-a268754b06db None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:23.372065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6486e5b9-e85f-4386-8acc-7fdac49205fa None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-ed6f9166-d9a9-48ba-a523-a268754b06db None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:23.372831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:23.373768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:23.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:23.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:23.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:23.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:23.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:23.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:23.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:23.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:23.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 31 00:55:23.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:23.378104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 448/882] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Jul 31 00:55:23 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:23.385055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6f9166-d9a9-48ba-a523-a268754b06db None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:23.414384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d237bd-c0e6-478f-816b-b2f65ba58174 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:23.415003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d237bd-c0e6-478f-816b-b2f65ba58174 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:23.415293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d237bd-c0e6-478f-816b-b2f65ba58174 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:23.415680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d237bd-c0e6-478f-816b-b2f65ba58174 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:23.416928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d237bd-c0e6-478f-816b-b2f65ba58174 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:23.418826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d237bd-c0e6-478f-816b-b2f65ba58174 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:23.447418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 435/883] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:55:23 2022] POST /identity/v3/services => generated 269 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:23.535089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6f9166-d9a9-48ba-a523-a268754b06db None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:23.552375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:23.598892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85e0ce9-9518-4787-9469-f8638a259e0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:23.599625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85e0ce9-9518-4787-9469-f8638a259e0c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:23.599900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85e0ce9-9518-4787-9469-f8638a259e0c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:23.600212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85e0ce9-9518-4787-9469-f8638a259e0c None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:23.601029 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85e0ce9-9518-4787-9469-f8638a259e0c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:23.601335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85e0ce9-9518-4787-9469-f8638a259e0c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:23.602810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85e0ce9-9518-4787-9469-f8638a259e0c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:23.625052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 449/884] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Jul 31 00:55:23 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:23.630592 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d237bd-c0e6-478f-816b-b2f65ba58174 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:23.682734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec481f56-cd3e-485f-9750-9e6aa4c97c4b None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:23.683243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec481f56-cd3e-485f-9750-9e6aa4c97c4b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:23.683243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec481f56-cd3e-485f-9750-9e6aa4c97c4b None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:23.685022 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 450/885] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:23.692185 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7d88a8-0628-4780-9a7a-2451cc25bacd None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:23.692490 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7d88a8-0628-4780-9a7a-2451cc25bacd None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:23.693341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7d88a8-0628-4780-9a7a-2451cc25bacd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:23.718076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-8e7d88a8-0628-4780-9a7a-2451cc25bacd None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:23.719285 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8e7d88a8-0628-4780-9a7a-2451cc25bacd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:23.742668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:23.752904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d237bd-c0e6-478f-816b-b2f65ba58174 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:23.798672 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3953e4-cb83-47f1-bb40-541526a0fa82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:23.799679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3953e4-cb83-47f1-bb40-541526a0fa82 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:23.799961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3953e4-cb83-47f1-bb40-541526a0fa82 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:23.800238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3953e4-cb83-47f1-bb40-541526a0fa82 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:23.822085 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 451/886] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:23 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:23.843568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7d88a8-0628-4780-9a7a-2451cc25bacd None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:23.896966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3953e4-cb83-47f1-bb40-541526a0fa82 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:23.898547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3953e4-cb83-47f1-bb40-541526a0fa82 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:23.943941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:23.956098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7d88a8-0628-4780-9a7a-2451cc25bacd None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:23.994058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac439a1b-1255-4fe7-9bc7-7dc63fbf0ed9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:23.994854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac439a1b-1255-4fe7-9bc7-7dc63fbf0ed9 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:23.995230 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac439a1b-1255-4fe7-9bc7-7dc63fbf0ed9 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:23.995624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac439a1b-1255-4fe7-9bc7-7dc63fbf0ed9 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:23.996366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac439a1b-1255-4fe7-9bc7-7dc63fbf0ed9 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:23.998108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac439a1b-1255-4fe7-9bc7-7dc63fbf0ed9 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-ac439a1b-1255-4fe7-9bc7-7dc63fbf0ed9 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:24.008143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:24.009243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:24.010077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 452/887] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Jul 31 00:55:23 2022] GET /identity/v3/services/object-store => generated 93 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:24.010869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99229773-0e37-4428-bf69-49ce5b5d10b4 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:24.010869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99229773-0e37-4428-bf69-49ce5b5d10b4 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:24.011660 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99229773-0e37-4428-bf69-49ce5b5d10b4 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:24.012446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 453/888] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:24 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:24.017680 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef330551-25ee-4fd0-886e-9415c6e794a8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:24.018025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef330551-25ee-4fd0-886e-9415c6e794a8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:24.018199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef330551-25ee-4fd0-886e-9415c6e794a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:24.029377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:24.038811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 436/889] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:23 2022] GET /identity/v3/auth/tokens => generated 1362 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:55:24.039487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3953e4-cb83-47f1-bb40-541526a0fa82 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:24.044098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ef330551-25ee-4fd0-886e-9415c6e794a8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:24.048752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ef330551-25ee-4fd0-886e-9415c6e794a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:24.127970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 454/890] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:24 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:24.128894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef330551-25ee-4fd0-886e-9415c6e794a8 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:24.201382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3953e4-cb83-47f1-bb40-541526a0fa82 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:24.241178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9809d6-b6f2-48c8-baa2-7790359b58de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:24.241807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9809d6-b6f2-48c8-baa2-7790359b58de None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:24.242088 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9809d6-b6f2-48c8-baa2-7790359b58de None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:24.242364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9809d6-b6f2-48c8-baa2-7790359b58de None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:24.243040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9809d6-b6f2-48c8-baa2-7790359b58de None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:24.243344 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9809d6-b6f2-48c8-baa2-7790359b58de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:24.244747 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9809d6-b6f2-48c8-baa2-7790359b58de None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:24.250421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:24.250593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 437/891] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Jul 31 00:55:24 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:24.251499 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9809d6-b6f2-48c8-baa2-7790359b58de None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:24.276973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef330551-25ee-4fd0-886e-9415c6e794a8 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:24.316961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:24.322023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1651c534-2445-41b1-824f-88f8711c11d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:24.322921 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1651c534-2445-41b1-824f-88f8711c11d9 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:24.323404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1651c534-2445-41b1-824f-88f8711c11d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:24.323886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1651c534-2445-41b1-824f-88f8711c11d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:24.335163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1651c534-2445-41b1-824f-88f8711c11d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:24.336913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1651c534-2445-41b1-824f-88f8711c11d9 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-1651c534-2445-41b1-824f-88f8711c11d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:24.353265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:24.354216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:24.355149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:24.355149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:24.355149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:24.355149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:24.356702 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 455/892] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:24.357687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1651c534-2445-41b1-824f-88f8711c11d9 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:24.370534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9809d6-b6f2-48c8-baa2-7790359b58de None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:24.411477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554904cd-4c38-4755-8835-45e1890b0228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:24.412179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554904cd-4c38-4755-8835-45e1890b0228 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:24.412453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554904cd-4c38-4755-8835-45e1890b0228 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:24.412732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554904cd-4c38-4755-8835-45e1890b0228 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:24.413409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554904cd-4c38-4755-8835-45e1890b0228 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:24.414804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554904cd-4c38-4755-8835-45e1890b0228 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-554904cd-4c38-4755-8835-45e1890b0228 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:24.428608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:24.429676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 31 00:55:24.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:24.435674 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 438/893] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:55:24 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:24.436665 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554904cd-4c38-4755-8835-45e1890b0228 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:24.482400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1651c534-2445-41b1-824f-88f8711c11d9 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:24.525506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c810b785-3955-4ad2-b76e-955ca3baccfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:24.525506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c810b785-3955-4ad2-b76e-955ca3baccfb None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:24.525506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c810b785-3955-4ad2-b76e-955ca3baccfb None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:24.525506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c810b785-3955-4ad2-b76e-955ca3baccfb None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:24.526109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c810b785-3955-4ad2-b76e-955ca3baccfb None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:24.526109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c810b785-3955-4ad2-b76e-955ca3baccfb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:24.526109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c810b785-3955-4ad2-b76e-955ca3baccfb None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:24.532417 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:24.547418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 456/894] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Jul 31 00:55:24 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:24.547418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c810b785-3955-4ad2-b76e-955ca3baccfb None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:24.574683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554904cd-4c38-4755-8835-45e1890b0228 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:24.608944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:24.625117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ae2b28-45e7-4757-923b-fcc94f18e3d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:24.626084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ae2b28-45e7-4757-923b-fcc94f18e3d9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:24.626457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ae2b28-45e7-4757-923b-fcc94f18e3d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:24.627118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ae2b28-45e7-4757-923b-fcc94f18e3d9 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:24.628186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ae2b28-45e7-4757-923b-fcc94f18e3d9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:24.628491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ae2b28-45e7-4757-923b-fcc94f18e3d9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:24.629910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ae2b28-45e7-4757-923b-fcc94f18e3d9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:24.648556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 439/895] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:24.652584 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ae2b28-45e7-4757-923b-fcc94f18e3d9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:24.685008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c810b785-3955-4ad2-b76e-955ca3baccfb None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:24.728005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdb0840-917d-4ba1-9d1a-37e0952cb831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:24.728986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdb0840-917d-4ba1-9d1a-37e0952cb831 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:24.729499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdb0840-917d-4ba1-9d1a-37e0952cb831 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:24.729771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdb0840-917d-4ba1-9d1a-37e0952cb831 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:24.730448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdb0840-917d-4ba1-9d1a-37e0952cb831 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:24.730784 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdb0840-917d-4ba1-9d1a-37e0952cb831 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:24.732182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdb0840-917d-4ba1-9d1a-37e0952cb831 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:24.738331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 457/896] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:55:24 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:24.745390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdb0840-917d-4ba1-9d1a-37e0952cb831 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:24.789230 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ae2b28-45e7-4757-923b-fcc94f18e3d9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:24.813546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:24.832858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de71c75-a457-4391-b03c-94c2ec5c5a37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:24.833504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de71c75-a457-4391-b03c-94c2ec5c5a37 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:24.833781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de71c75-a457-4391-b03c-94c2ec5c5a37 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:24.834052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de71c75-a457-4391-b03c-94c2ec5c5a37 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:24.842013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de71c75-a457-4391-b03c-94c2ec5c5a37 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:24.843626 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de71c75-a457-4391-b03c-94c2ec5c5a37 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-0de71c75-a457-4391-b03c-94c2ec5c5a37 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:24.855909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:24.856823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:24.857693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:24.857693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:24.857693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:24.857693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:24.857693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:24.857693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:24.857693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:24.857693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:24.858743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 440/897] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:24.861541 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdb0840-917d-4ba1-9d1a-37e0952cb831 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:24.863860 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de71c75-a457-4391-b03c-94c2ec5c5a37 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:24.900917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2df6c53-fe53-4a18-9490-b06a86278d91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:24.901907 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2df6c53-fe53-4a18-9490-b06a86278d91 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:24.902411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2df6c53-fe53-4a18-9490-b06a86278d91 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:24.902897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2df6c53-fe53-4a18-9490-b06a86278d91 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:24.903836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2df6c53-fe53-4a18-9490-b06a86278d91 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:24.904368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2df6c53-fe53-4a18-9490-b06a86278d91 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:24.906090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2df6c53-fe53-4a18-9490-b06a86278d91 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:24.916599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 458/898] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:55:24 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:24.918123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2df6c53-fe53-4a18-9490-b06a86278d91 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:24.918439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:24.979976 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de71c75-a457-4391-b03c-94c2ec5c5a37 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:25.019262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fa3ff4-45a4-4509-88af-bf7c2ec1bfda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:25.019894 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fa3ff4-45a4-4509-88af-bf7c2ec1bfda None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:25.020387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fa3ff4-45a4-4509-88af-bf7c2ec1bfda None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:25.020504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fa3ff4-45a4-4509-88af-bf7c2ec1bfda None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:25.021243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fa3ff4-45a4-4509-88af-bf7c2ec1bfda None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:25.021637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fa3ff4-45a4-4509-88af-bf7c2ec1bfda None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:25.023007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fa3ff4-45a4-4509-88af-bf7c2ec1bfda None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:25.030327 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2df6c53-fe53-4a18-9490-b06a86278d91 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:25.039449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 441/899] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:25.044933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fa3ff4-45a4-4509-88af-bf7c2ec1bfda None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:25.059911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:25.069036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b598d652-50e0-4454-91be-199cce6c3240 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:25.070027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b598d652-50e0-4454-91be-199cce6c3240 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:25.070510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b598d652-50e0-4454-91be-199cce6c3240 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:25.071018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b598d652-50e0-4454-91be-199cce6c3240 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:25.072024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b598d652-50e0-4454-91be-199cce6c3240 None admin] RBAC: Adding query filter params (interface=public, service_id=65513a4094a64caabdf0bd51f0781c69, region_id=RegionOne) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:25.072555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b598d652-50e0-4454-91be-199cce6c3240 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:25.074224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b598d652-50e0-4454-91be-199cce6c3240 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:25.087613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 459/900] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Jul 31 00:55:24 2022] GET /identity/v3/endpoints?service_id=65513a4094a64caabdf0bd51f0781c69&interface=public®ion_id=RegionOne => generated 194 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:25.089179 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b598d652-50e0-4454-91be-199cce6c3240 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:25.158611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fa3ff4-45a4-4509-88af-bf7c2ec1bfda None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:25.161226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:25.196743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b598d652-50e0-4454-91be-199cce6c3240 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:25.202889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6624c8-d8b2-46b7-92fc-05ee6e25c2d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:25.203544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6624c8-d8b2-46b7-92fc-05ee6e25c2d4 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:25.203824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6624c8-d8b2-46b7-92fc-05ee6e25c2d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:25.204098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6624c8-d8b2-46b7-92fc-05ee6e25c2d4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:25.204889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6624c8-d8b2-46b7-92fc-05ee6e25c2d4 None admin] RBAC: Adding query filter params (role.id=53d929a69ffa476eac93c56d7de72a16, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=8cdffe91785e42d798b931082969598e) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:25.205220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6624c8-d8b2-46b7-92fc-05ee6e25c2d4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:25.206631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6624c8-d8b2-46b7-92fc-05ee6e25c2d4 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:25.213369 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 442/901] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:25 2022] GET /identity/v3/role_assignments?user.id=8cdffe91785e42d798b931082969598e&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=53d929a69ffa476eac93c56d7de72a16 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:25.239911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1108d9b1-bf84-44a6-a303-c152efa9d152 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:25.240980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1108d9b1-bf84-44a6-a303-c152efa9d152 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:25.241488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1108d9b1-bf84-44a6-a303-c152efa9d152 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:25.242044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1108d9b1-bf84-44a6-a303-c152efa9d152 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:25.242958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1108d9b1-bf84-44a6-a303-c152efa9d152 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:25.244635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1108d9b1-bf84-44a6-a303-c152efa9d152 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:25.259036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 460/902] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Jul 31 00:55:25 2022] GET /identity/v3/services => generated 1268 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 00:55:25.633652 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa52d88-d500-42be-a52f-36aff8508b9f None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:25.634632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa52d88-d500-42be-a52f-36aff8508b9f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:25.635130 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa52d88-d500-42be-a52f-36aff8508b9f None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:25.637098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 443/903] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:25.643228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d963c8c4-63c1-4d8c-9d13-845921a2c7b7 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:25.643888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d963c8c4-63c1-4d8c-9d13-845921a2c7b7 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:25.644363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d963c8c4-63c1-4d8c-9d13-845921a2c7b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:25.664626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-d963c8c4-63c1-4d8c-9d13-845921a2c7b7 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:25.666188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d963c8c4-63c1-4d8c-9d13-845921a2c7b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:25.729169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:25.753130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 461/904] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:25 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:25.775934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa52d88-d500-42be-a52f-36aff8508b9f None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:25.919692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa52d88-d500-42be-a52f-36aff8508b9f None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:25.957402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcd159f-8624-4e6c-aaf2-4553f8ef2a65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:25.958054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcd159f-8624-4e6c-aaf2-4553f8ef2a65 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:25.958333 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcd159f-8624-4e6c-aaf2-4553f8ef2a65 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:25.958607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcd159f-8624-4e6c-aaf2-4553f8ef2a65 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:25.959236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcd159f-8624-4e6c-aaf2-4553f8ef2a65 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:25.960642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcd159f-8624-4e6c-aaf2-4553f8ef2a65 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-fbcd159f-8624-4e6c-aaf2-4553f8ef2a65 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:25.970781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:25.971785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:25.973033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 444/905] 10.222.0.42 () {56 vars in 1132 bytes} [Sun Jul 31 00:55:25 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:25.980503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d963c8c4-63c1-4d8c-9d13-845921a2c7b7 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:26.118304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d963c8c4-63c1-4d8c-9d13-845921a2c7b7 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:26.158234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4754e6a-1186-4b61-9bcd-fbe98c218c34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:26.159298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4754e6a-1186-4b61-9bcd-fbe98c218c34 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:26.159840 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4754e6a-1186-4b61-9bcd-fbe98c218c34 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:26.160335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4754e6a-1186-4b61-9bcd-fbe98c218c34 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:26.161594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4754e6a-1186-4b61-9bcd-fbe98c218c34 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:26.162159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4754e6a-1186-4b61-9bcd-fbe98c218c34 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:26.164715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4754e6a-1186-4b61-9bcd-fbe98c218c34 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:26.170874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 462/906] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Jul 31 00:55:25 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:26.176779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcd159f-8624-4e6c-aaf2-4553f8ef2a65 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:26.226155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:26.325238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcd159f-8624-4e6c-aaf2-4553f8ef2a65 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:26.363823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678cac6-9251-4d88-b361-a99c2244548e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:26.364446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678cac6-9251-4d88-b361-a99c2244548e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:26.364724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678cac6-9251-4d88-b361-a99c2244548e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:26.365003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678cac6-9251-4d88-b361-a99c2244548e None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:26.365715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c678cac6-9251-4d88-b361-a99c2244548e None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:26.366020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c678cac6-9251-4d88-b361-a99c2244548e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:26.367479 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c678cac6-9251-4d88-b361-a99c2244548e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:26.373837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 445/907] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Jul 31 00:55:26 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:26.380565 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4754e6a-1186-4b61-9bcd-fbe98c218c34 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:26.435501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:26.490900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4754e6a-1186-4b61-9bcd-fbe98c218c34 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:26.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbd6b03-d413-4166-bb05-0cc0a08675b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:26.532919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbd6b03-d413-4166-bb05-0cc0a08675b6 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:26.533650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbd6b03-d413-4166-bb05-0cc0a08675b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:26.534311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbd6b03-d413-4166-bb05-0cc0a08675b6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:26.535785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbd6b03-d413-4166-bb05-0cc0a08675b6 None admin] RBAC: Adding query filter params (interface=public, service_id=615899e367c7477ea67b210646cf8246, region_id=RegionOne) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:26.536607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbd6b03-d413-4166-bb05-0cc0a08675b6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:26.539269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbd6b03-d413-4166-bb05-0cc0a08675b6 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:26.546534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 463/908] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Jul 31 00:55:26 2022] GET /identity/v3/endpoints?service_id=615899e367c7477ea67b210646cf8246&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:26.553084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678cac6-9251-4d88-b361-a99c2244548e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:26.692615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678cac6-9251-4d88-b361-a99c2244548e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:26.700529 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:26.734667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d819fd-8805-4114-96a5-86ba25003d78 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:26.735888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d819fd-8805-4114-96a5-86ba25003d78 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:26.735888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d819fd-8805-4114-96a5-86ba25003d78 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:26.736806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5077f0a8-61fb-4cef-8fe4-72a8c9bcadb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:26.737615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5077f0a8-61fb-4cef-8fe4-72a8c9bcadb9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:26.738043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5077f0a8-61fb-4cef-8fe4-72a8c9bcadb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:26.738253 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 464/909] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:55:26.738828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5077f0a8-61fb-4cef-8fe4-72a8c9bcadb9 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:26.739750 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5077f0a8-61fb-4cef-8fe4-72a8c9bcadb9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:26.741434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5077f0a8-61fb-4cef-8fe4-72a8c9bcadb9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:26.742853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244e2817-fe26-45ae-9068-47d82b4a263f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:26.743167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244e2817-fe26-45ae-9068-47d82b4a263f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:26.743507 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244e2817-fe26-45ae-9068-47d82b4a263f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:26.749694 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 446/910] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Jul 31 00:55:26 2022] GET /identity/v3/services => generated 1268 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 00:55:26.765627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-244e2817-fe26-45ae-9068-47d82b4a263f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:26.766911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-244e2817-fe26-45ae-9068-47d82b4a263f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:26.854108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 465/911] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:26 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:26.877124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5077f0a8-61fb-4cef-8fe4-72a8c9bcadb9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:26.994397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5077f0a8-61fb-4cef-8fe4-72a8c9bcadb9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:27.038033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81637c0-7774-4562-b0a2-5209ba7e702c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:27.039065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81637c0-7774-4562-b0a2-5209ba7e702c None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:27.039624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81637c0-7774-4562-b0a2-5209ba7e702c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:27.040090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81637c0-7774-4562-b0a2-5209ba7e702c None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:27.041277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81637c0-7774-4562-b0a2-5209ba7e702c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:27.043882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81637c0-7774-4562-b0a2-5209ba7e702c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:27.077556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 447/912] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:55:26 2022] POST /identity/v3/services => generated 241 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:27.334080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0977d75c-1b65-4349-9ccd-b75c1b92f947 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:27.334268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0977d75c-1b65-4349-9ccd-b75c1b92f947 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:27.334547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0977d75c-1b65-4349-9ccd-b75c1b92f947 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:27.336189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 466/913] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:27.341899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6675ce83-5837-4f72-a100-7acf3c31a6e5 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:27.342186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6675ce83-5837-4f72-a100-7acf3c31a6e5 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:27.342450 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6675ce83-5837-4f72-a100-7acf3c31a6e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:27.368224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6675ce83-5837-4f72-a100-7acf3c31a6e5 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:27.370163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6675ce83-5837-4f72-a100-7acf3c31a6e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:27.393745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:27.508002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 448/914] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:27 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:27.544182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d689e0-9a22-41bd-a4e7-b13f6ad6069b None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:27.544596 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d689e0-9a22-41bd-a4e7-b13f6ad6069b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:27.544743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d689e0-9a22-41bd-a4e7-b13f6ad6069b None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:27.546474 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 467/915] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:27.554001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ffd4be-3607-442c-9fad-724e6fc1fc20 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:27.554462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ffd4be-3607-442c-9fad-724e6fc1fc20 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:27.554717 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d689e0-9a22-41bd-a4e7-b13f6ad6069b None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:27.554952 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ffd4be-3607-442c-9fad-724e6fc1fc20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:27.587149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-e8ffd4be-3607-442c-9fad-724e6fc1fc20 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:27.588371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e8ffd4be-3607-442c-9fad-724e6fc1fc20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:27.653162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:27.706802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 449/916] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:27 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:27.725834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d689e0-9a22-41bd-a4e7-b13f6ad6069b None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:27.730983 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ffd4be-3607-442c-9fad-724e6fc1fc20 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:27.771280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a1801b-4a31-4cfd-9448-35cb04c0c1dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:27.772716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a1801b-4a31-4cfd-9448-35cb04c0c1dd None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:27.772997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a1801b-4a31-4cfd-9448-35cb04c0c1dd None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:27.773269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a1801b-4a31-4cfd-9448-35cb04c0c1dd None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:27.797785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a1801b-4a31-4cfd-9448-35cb04c0c1dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-88a1801b-4a31-4cfd-9448-35cb04c0c1dd None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:27.806304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:27.807813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:27.808661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:27.808661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:27.808661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:27.808661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:27.808661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:27.808661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:27.808661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:27.808661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:27.808661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:27.808661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:27.808661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 31 00:55:27.808661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:27.817554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 468/917] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Jul 31 00:55:27 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:27.821882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a1801b-4a31-4cfd-9448-35cb04c0c1dd None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:27.824253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:27.854955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ffd4be-3607-442c-9fad-724e6fc1fc20 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:27.892705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc765c0a-d854-426e-a1c5-77c71e0159b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:27.893369 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc765c0a-d854-426e-a1c5-77c71e0159b5 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:27.893649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc765c0a-d854-426e-a1c5-77c71e0159b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:27.893922 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc765c0a-d854-426e-a1c5-77c71e0159b5 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:27.894556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc765c0a-d854-426e-a1c5-77c71e0159b5 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:27.895991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc765c0a-d854-426e-a1c5-77c71e0159b5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-bc765c0a-d854-426e-a1c5-77c71e0159b5 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:27.906501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:27.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 31 00:55:27.911133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:27.913307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 450/918] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:55:27 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:27.914475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc765c0a-d854-426e-a1c5-77c71e0159b5 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:28.019234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a1801b-4a31-4cfd-9448-35cb04c0c1dd None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:28.023489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc765c0a-d854-426e-a1c5-77c71e0159b5 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:28.024510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:28.062923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccedd40-e0f3-4c67-8360-ded4e6f8ac36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:28.063587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccedd40-e0f3-4c67-8360-ded4e6f8ac36 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:28.063870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccedd40-e0f3-4c67-8360-ded4e6f8ac36 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:28.064148 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccedd40-e0f3-4c67-8360-ded4e6f8ac36 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:28.064938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccedd40-e0f3-4c67-8360-ded4e6f8ac36 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:28.065244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccedd40-e0f3-4c67-8360-ded4e6f8ac36 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:28.066731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccedd40-e0f3-4c67-8360-ded4e6f8ac36 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:28.071668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5072aba5-9f26-4814-a6d8-a1ee2488580b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:28.072618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5072aba5-9f26-4814-a6d8-a1ee2488580b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:28.073103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5072aba5-9f26-4814-a6d8-a1ee2488580b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:28.073579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5072aba5-9f26-4814-a6d8-a1ee2488580b None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:28.074627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5072aba5-9f26-4814-a6d8-a1ee2488580b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:28.075141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5072aba5-9f26-4814-a6d8-a1ee2488580b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:28.076794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5072aba5-9f26-4814-a6d8-a1ee2488580b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:28.077833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 451/919] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:55:27 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:28.078793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccedd40-e0f3-4c67-8360-ded4e6f8ac36 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:28.082849 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:28.095345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 469/920] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Jul 31 00:55:27 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:28.097273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5072aba5-9f26-4814-a6d8-a1ee2488580b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:28.197105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccedd40-e0f3-4c67-8360-ded4e6f8ac36 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:28.219124 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5072aba5-9f26-4814-a6d8-a1ee2488580b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:28.235893 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d02bef0-e3fa-4546-8fe3-cfd2767a6687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:28.236627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d02bef0-e3fa-4546-8fe3-cfd2767a6687 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:28.236846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d02bef0-e3fa-4546-8fe3-cfd2767a6687 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:28.237130 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d02bef0-e3fa-4546-8fe3-cfd2767a6687 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:28.237811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d02bef0-e3fa-4546-8fe3-cfd2767a6687 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:28.238114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d02bef0-e3fa-4546-8fe3-cfd2767a6687 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:28.239634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d02bef0-e3fa-4546-8fe3-cfd2767a6687 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:28.249863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 452/921] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Jul 31 00:55:28 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:28.250855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d02bef0-e3fa-4546-8fe3-cfd2767a6687 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:28.259167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c31bb37-3a5c-4a43-86e3-d0cbae472279 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:28.260202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c31bb37-3a5c-4a43-86e3-d0cbae472279 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:28.260784 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c31bb37-3a5c-4a43-86e3-d0cbae472279 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:28.261260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c31bb37-3a5c-4a43-86e3-d0cbae472279 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:28.274125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c31bb37-3a5c-4a43-86e3-d0cbae472279 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:28.276113 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c31bb37-3a5c-4a43-86e3-d0cbae472279 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-6c31bb37-3a5c-4a43-86e3-d0cbae472279 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:28.292870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:28.293790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:28.294740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:28.294740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:28.294740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:28.294740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:28.301823 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 470/922] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:28.303235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c31bb37-3a5c-4a43-86e3-d0cbae472279 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:28.324709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:28.326901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:28.381853 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d02bef0-e3fa-4546-8fe3-cfd2767a6687 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:28.443483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c31bb37-3a5c-4a43-86e3-d0cbae472279 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:28.445825 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8854c34-872e-4253-bbf3-ed7fa58d65e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:28.447031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8854c34-872e-4253-bbf3-ed7fa58d65e9 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:28.447416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8854c34-872e-4253-bbf3-ed7fa58d65e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:28.447819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8854c34-872e-4253-bbf3-ed7fa58d65e9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:28.449018 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8854c34-872e-4253-bbf3-ed7fa58d65e9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:28.451238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8854c34-872e-4253-bbf3-ed7fa58d65e9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:28.483069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d333ed7b-3d5a-443c-bcff-531512ca5586 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:28.484092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d333ed7b-3d5a-443c-bcff-531512ca5586 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:28.484520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d333ed7b-3d5a-443c-bcff-531512ca5586 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:28.484945 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d333ed7b-3d5a-443c-bcff-531512ca5586 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:28.485824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d333ed7b-3d5a-443c-bcff-531512ca5586 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:28.486264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d333ed7b-3d5a-443c-bcff-531512ca5586 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:28.487789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 453/923] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 00:55:28 2022] POST /identity/v3/endpoints => generated 346 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:28.488142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d333ed7b-3d5a-443c-bcff-531512ca5586 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:28.507531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 471/924] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:28.513620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8854c34-872e-4253-bbf3-ed7fa58d65e9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:28.705650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8854c34-872e-4253-bbf3-ed7fa58d65e9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:28.739487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:28.768642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a542b46-6d88-4aef-bf24-ce1e522853a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:28.770081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a542b46-6d88-4aef-bf24-ce1e522853a3 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:28.770775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a542b46-6d88-4aef-bf24-ce1e522853a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:28.771496 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a542b46-6d88-4aef-bf24-ce1e522853a3 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:28.787017 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a542b46-6d88-4aef-bf24-ce1e522853a3 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:28.790254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a542b46-6d88-4aef-bf24-ce1e522853a3 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-9a542b46-6d88-4aef-bf24-ce1e522853a3 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:28.815285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:28.817383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:28.819561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:28.819561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:28.819561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:28.819561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:28.819561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:28.819561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:28.819561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:28.819561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:28.819561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 31 00:55:28.819561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:28.823822 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 454/925] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Jul 31 00:55:28 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:28.832783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d333ed7b-3d5a-443c-bcff-531512ca5586 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:28.927868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:28.953596 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d333ed7b-3d5a-443c-bcff-531512ca5586 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:28.991404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1472e2c-cba4-4780-aa1d-a26d7d05d164 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:28.992329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1472e2c-cba4-4780-aa1d-a26d7d05d164 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:28.992769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1472e2c-cba4-4780-aa1d-a26d7d05d164 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:28.993198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1472e2c-cba4-4780-aa1d-a26d7d05d164 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:28.994250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1472e2c-cba4-4780-aa1d-a26d7d05d164 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:28.994705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1472e2c-cba4-4780-aa1d-a26d7d05d164 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:28.996335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1472e2c-cba4-4780-aa1d-a26d7d05d164 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:29.017879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 472/926] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Jul 31 00:55:28 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:29.024202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a542b46-6d88-4aef-bf24-ce1e522853a3 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:29.141161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a542b46-6d88-4aef-bf24-ce1e522853a3 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:29.178303 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38493ca9-f10e-4265-b47f-a520127f057a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:29.178955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38493ca9-f10e-4265-b47f-a520127f057a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:29.179235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38493ca9-f10e-4265-b47f-a520127f057a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:29.179536 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38493ca9-f10e-4265-b47f-a520127f057a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:29.196103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38493ca9-f10e-4265-b47f-a520127f057a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:29.199070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38493ca9-f10e-4265-b47f-a520127f057a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-38493ca9-f10e-4265-b47f-a520127f057a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:29.214834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:29.216629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:29.218890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:29.218890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:29.218890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:29.218890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:29.218890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 455/927] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:29.222615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1472e2c-cba4-4780-aa1d-a26d7d05d164 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:29.337495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1472e2c-cba4-4780-aa1d-a26d7d05d164 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:29.368599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0810213-04cc-46c4-82e0-8d7ebf7fd7ba None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:29.368885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0810213-04cc-46c4-82e0-8d7ebf7fd7ba None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:29.369167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0810213-04cc-46c4-82e0-8d7ebf7fd7ba None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:29.370931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 456/928] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:29.377485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856cf323-be9b-4dca-86cb-d0d56ac53685 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:29.378068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856cf323-be9b-4dca-86cb-d0d56ac53685 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:29.378573 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856cf323-be9b-4dca-86cb-d0d56ac53685 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:29.402536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959efbed-5749-4daf-afbe-def20930e768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:29.403596 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959efbed-5749-4daf-afbe-def20930e768 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:29.404087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959efbed-5749-4daf-afbe-def20930e768 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:29.404630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959efbed-5749-4daf-afbe-def20930e768 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:29.406454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959efbed-5749-4daf-afbe-def20930e768 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:29.406983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959efbed-5749-4daf-afbe-def20930e768 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:29.409825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959efbed-5749-4daf-afbe-def20930e768 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:29.410991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-856cf323-be9b-4dca-86cb-d0d56ac53685 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:29.413467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-856cf323-be9b-4dca-86cb-d0d56ac53685 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:29.414970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:29.443465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 473/929] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:29.446593 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959efbed-5749-4daf-afbe-def20930e768 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:29.493561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:29.577669 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 457/930] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:29 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:29.579637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58833be-825c-48a8-a5cd-705d10f7c109 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:29.579984 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58833be-825c-48a8-a5cd-705d10f7c109 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:29.580313 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58833be-825c-48a8-a5cd-705d10f7c109 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:29.581862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 458/931] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:29.595494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d178d5d2-6dc3-4aea-9087-4017ae108732 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:29.595494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d178d5d2-6dc3-4aea-9087-4017ae108732 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:29.595494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d178d5d2-6dc3-4aea-9087-4017ae108732 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:29.621215 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d178d5d2-6dc3-4aea-9087-4017ae108732 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:29.623640 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d178d5d2-6dc3-4aea-9087-4017ae108732 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:29.654818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959efbed-5749-4daf-afbe-def20930e768 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:29.700509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a14e6bb-0b3b-4965-ba1d-f4ede3a84e55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:29.701310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a14e6bb-0b3b-4965-ba1d-f4ede3a84e55 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:29.701693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a14e6bb-0b3b-4965-ba1d-f4ede3a84e55 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:29.702612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a14e6bb-0b3b-4965-ba1d-f4ede3a84e55 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:29.707402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 459/932] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:29 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:29.707402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d178d5d2-6dc3-4aea-9087-4017ae108732 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:29.717220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a14e6bb-0b3b-4965-ba1d-f4ede3a84e55 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:29.719687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a14e6bb-0b3b-4965-ba1d-f4ede3a84e55 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-6a14e6bb-0b3b-4965-ba1d-f4ede3a84e55 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:29.732901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:29.733892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:29.734777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:29.734777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:29.734777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:29.734777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:29.734777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:29.734777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:29.734777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:29.734777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:29.737406 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 474/933] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:29.746025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a14e6bb-0b3b-4965-ba1d-f4ede3a84e55 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:29.830246 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:29.863483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:29.881207 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d178d5d2-6dc3-4aea-9087-4017ae108732 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:29.903038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a14e6bb-0b3b-4965-ba1d-f4ede3a84e55 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:29.921911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2c55b5-6637-4790-b6bd-e74905cd411b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:29.922627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2c55b5-6637-4790-b6bd-e74905cd411b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:29.922911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2c55b5-6637-4790-b6bd-e74905cd411b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:29.923167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2c55b5-6637-4790-b6bd-e74905cd411b None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:29.924541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2c55b5-6637-4790-b6bd-e74905cd411b None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:29.926179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2c55b5-6637-4790-b6bd-e74905cd411b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-5b2c55b5-6637-4790-b6bd-e74905cd411b None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:29.942722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:29.944054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:29.944887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 460/934] 10.222.0.42 () {56 vars in 1132 bytes} [Sun Jul 31 00:55:29 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:29.946286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2c55b5-6637-4790-b6bd-e74905cd411b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:29.978000 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1903793e-4b6b-4426-bbb0-2ccd1552c8af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:29.979033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1903793e-4b6b-4426-bbb0-2ccd1552c8af None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:29.980156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1903793e-4b6b-4426-bbb0-2ccd1552c8af None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:29.980722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1903793e-4b6b-4426-bbb0-2ccd1552c8af None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:29.982055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1903793e-4b6b-4426-bbb0-2ccd1552c8af None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:29.982695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1903793e-4b6b-4426-bbb0-2ccd1552c8af None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:29.985209 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1903793e-4b6b-4426-bbb0-2ccd1552c8af None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:30.007881 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 475/935] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:30.009275 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1903793e-4b6b-4426-bbb0-2ccd1552c8af None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:30.117957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2c55b5-6637-4790-b6bd-e74905cd411b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:30.177845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2e693c-bd6c-4e58-a51f-ef9d921a7398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:30.178474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2e693c-bd6c-4e58-a51f-ef9d921a7398 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:30.181172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2e693c-bd6c-4e58-a51f-ef9d921a7398 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:30.181172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2e693c-bd6c-4e58-a51f-ef9d921a7398 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:30.181172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2e693c-bd6c-4e58-a51f-ef9d921a7398 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:30.181172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2e693c-bd6c-4e58-a51f-ef9d921a7398 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-6f2e693c-bd6c-4e58-a51f-ef9d921a7398 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:30.194931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:30.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:30.198574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 461/936] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Jul 31 00:55:29 2022] GET /identity/v3/services/object-store => generated 93 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:30.202876 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2e693c-bd6c-4e58-a51f-ef9d921a7398 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:30.209779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1903793e-4b6b-4426-bbb0-2ccd1552c8af None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:30.236178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:30.240620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:30.269469 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd6ed5c-1e3a-4bc4-8466-ed7bebfaa21f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:30.270363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd6ed5c-1e3a-4bc4-8466-ed7bebfaa21f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:30.271615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd6ed5c-1e3a-4bc4-8466-ed7bebfaa21f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:30.271615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd6ed5c-1e3a-4bc4-8466-ed7bebfaa21f None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:30.272137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd6ed5c-1e3a-4bc4-8466-ed7bebfaa21f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:30.272411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd6ed5c-1e3a-4bc4-8466-ed7bebfaa21f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:30.273861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd6ed5c-1e3a-4bc4-8466-ed7bebfaa21f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:30.282222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 476/937] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Jul 31 00:55:30 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:30.283425 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd6ed5c-1e3a-4bc4-8466-ed7bebfaa21f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:30.384342 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2e693c-bd6c-4e58-a51f-ef9d921a7398 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:30.455743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae36414e-d1f2-4c68-8721-bb815c10304d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:30.456891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae36414e-d1f2-4c68-8721-bb815c10304d None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:30.457384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae36414e-d1f2-4c68-8721-bb815c10304d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:30.457563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd6ed5c-1e3a-4bc4-8466-ed7bebfaa21f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:30.457880 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae36414e-d1f2-4c68-8721-bb815c10304d None admin] PATH_INFO: `/v3/projects/8b41573c9f3146d19a306089d686015c/users/8cdffe91785e42d798b931082969598e/roles/53d929a69ffa476eac93c56d7de72a16` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:30.461805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae36414e-d1f2-4c68-8721-bb815c10304d None admin] RBAC: Authorizing `identity:create_grant(project_id=8b41573c9f3146d19a306089d686015c, user_id=8cdffe91785e42d798b931082969598e, role_id=53d929a69ffa476eac93c56d7de72a16)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:30.464560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae36414e-d1f2-4c68-8721-bb815c10304d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:30.489213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 462/938] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:55:30 2022] PUT /identity/v3/projects/8b41573c9f3146d19a306089d686015c/users/8cdffe91785e42d798b931082969598e/roles/53d929a69ffa476eac93c56d7de72a16 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:55:30.490705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae36414e-d1f2-4c68-8721-bb815c10304d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:30.531595 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963b9b8a-b3bc-40ac-99e3-61a552a0b151 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:30.532909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963b9b8a-b3bc-40ac-99e3-61a552a0b151 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:30.534557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963b9b8a-b3bc-40ac-99e3-61a552a0b151 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:30.535089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963b9b8a-b3bc-40ac-99e3-61a552a0b151 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:30.536752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963b9b8a-b3bc-40ac-99e3-61a552a0b151 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:30.539186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963b9b8a-b3bc-40ac-99e3-61a552a0b151 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:30.547762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963b9b8a-b3bc-40ac-99e3-61a552a0b151 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:30.552810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 477/939] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Jul 31 00:55:30 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:30.560848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963b9b8a-b3bc-40ac-99e3-61a552a0b151 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:30.651559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:30.724790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:30.775074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae36414e-d1f2-4c68-8721-bb815c10304d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:30.788854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963b9b8a-b3bc-40ac-99e3-61a552a0b151 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:30.832360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a6de91-a030-4692-b0d0-83f7169d3a12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:30.833823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a6de91-a030-4692-b0d0-83f7169d3a12 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:30.834597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a6de91-a030-4692-b0d0-83f7169d3a12 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:30.835384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a6de91-a030-4692-b0d0-83f7169d3a12 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:30.836936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a6de91-a030-4692-b0d0-83f7169d3a12 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:30.837791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a6de91-a030-4692-b0d0-83f7169d3a12 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:30.839994 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a6de91-a030-4692-b0d0-83f7169d3a12 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:30.846808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 463/940] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Jul 31 00:55:30 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 357 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:30.864068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a6de91-a030-4692-b0d0-83f7169d3a12 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:30.880358 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1ec0ff-23d8-4c5f-8a3c-68549d90fba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:30.881387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1ec0ff-23d8-4c5f-8a3c-68549d90fba4 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:30.881727 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1ec0ff-23d8-4c5f-8a3c-68549d90fba4 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:30.882497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1ec0ff-23d8-4c5f-8a3c-68549d90fba4 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:30.883468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1ec0ff-23d8-4c5f-8a3c-68549d90fba4 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:30.883916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1ec0ff-23d8-4c5f-8a3c-68549d90fba4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:30.885557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1ec0ff-23d8-4c5f-8a3c-68549d90fba4 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:30.902365 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 478/941] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Jul 31 00:55:30 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 338 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:30.908580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1ec0ff-23d8-4c5f-8a3c-68549d90fba4 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:31.057027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a6de91-a030-4692-b0d0-83f7169d3a12 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:31.081628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1ec0ff-23d8-4c5f-8a3c-68549d90fba4 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:31.097924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:31.113492 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bb494a-3225-4a93-9858-c6445c62a049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:31.114289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bb494a-3225-4a93-9858-c6445c62a049 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:31.114688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bb494a-3225-4a93-9858-c6445c62a049 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:31.115081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bb494a-3225-4a93-9858-c6445c62a049 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:31.116241 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bb494a-3225-4a93-9858-c6445c62a049 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:31.117934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bb494a-3225-4a93-9858-c6445c62a049 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:31.133425 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b832f69-bb30-4f1a-95ad-8b4d9e76d9e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:31.135105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b832f69-bb30-4f1a-95ad-8b4d9e76d9e7 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:31.135105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b832f69-bb30-4f1a-95ad-8b4d9e76d9e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:31.135105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b832f69-bb30-4f1a-95ad-8b4d9e76d9e7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:31.135590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b832f69-bb30-4f1a-95ad-8b4d9e76d9e7 None admin] RBAC: Adding query filter params (interface=public, service_id=975b02a2f0824e02bf7ba610a874756e, region_id=RegionOne) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:31.135740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b832f69-bb30-4f1a-95ad-8b4d9e76d9e7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:31.137041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b832f69-bb30-4f1a-95ad-8b4d9e76d9e7 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:31.142863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:31.148887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 479/942] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Jul 31 00:55:30 2022] GET /identity/v3/endpoints?service_id=975b02a2f0824e02bf7ba610a874756e&interface=public®ion_id=RegionOne => generated 194 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:31.148887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1962527b-9e56-4d64-9a43-6248b356c4fa None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:31.148887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1962527b-9e56-4d64-9a43-6248b356c4fa None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:31.148887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1962527b-9e56-4d64-9a43-6248b356c4fa None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:31.148887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 480/943] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:31.151406 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1962527b-9e56-4d64-9a43-6248b356c4fa None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:31.156619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 464/944] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 00:55:30 2022] POST /identity/v3/endpoints => generated 347 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 00:55:31.158342 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c6722e-3c0b-4a29-ab3d-50260e59c333 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:31.158909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c6722e-3c0b-4a29-ab3d-50260e59c333 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:31.160219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c6722e-3c0b-4a29-ab3d-50260e59c333 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:31.185735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-29c6722e-3c0b-4a29-ab3d-50260e59c333 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:31.187760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-29c6722e-3c0b-4a29-ab3d-50260e59c333 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:31.296408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 465/945] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:31 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:31.312521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c6722e-3c0b-4a29-ab3d-50260e59c333 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:31.362812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1962527b-9e56-4d64-9a43-6248b356c4fa None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:31.433391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca01c65a-18bd-40e3-a523-0045129c10ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:31.433391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca01c65a-18bd-40e3-a523-0045129c10ca None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:31.433391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca01c65a-18bd-40e3-a523-0045129c10ca None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:31.433391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca01c65a-18bd-40e3-a523-0045129c10ca None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:31.434076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca01c65a-18bd-40e3-a523-0045129c10ca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:31.435429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca01c65a-18bd-40e3-a523-0045129c10ca None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:31.443637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 481/946] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Jul 31 00:55:31 2022] GET /identity/v3/services => generated 1497 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 00:55:31.470648 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c6722e-3c0b-4a29-ab3d-50260e59c333 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:31.505172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:31.528971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cae040-3822-4d97-ae92-51b372986e71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:31.530408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cae040-3822-4d97-ae92-51b372986e71 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:31.531090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cae040-3822-4d97-ae92-51b372986e71 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:31.531882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cae040-3822-4d97-ae92-51b372986e71 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:31.533505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67cae040-3822-4d97-ae92-51b372986e71 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:31.536623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67cae040-3822-4d97-ae92-51b372986e71 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:31.541790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: WARNING keystone.server.flask.common [None req-67cae040-3822-4d97-ae92-51b372986e71 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 31 00:55:31.574275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.sql.core [None req-67cae040-3822-4d97-ae92-51b372986e71 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 31 00:55:31.574275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 31 00:55:31.574275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [parameters: {'id': 'dd1a476d188d451283954eb3cfc89829', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 31 00:55:31.574275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80208) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-67cae040-3822-4d97-ae92-51b372986e71 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id dd1a476d188d451283954eb3cfc89829.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id dd1a476d188d451283954eb3cfc89829. Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application result = self._query(query) Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application conn.query(q) Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 31 00:55:31.583673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application result.read() Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Jul 31 00:55:31.585318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return project_ref.to_dict() Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application next(self.gen) Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application yield resource Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application next(self.gen) Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self.session.rollback() Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self.force_reraise() Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise self.value Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application session.commit() Jul 31 00:55:31.587469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._prepare_impl() Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self.session.flush() Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._flush(objects) Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application compat.raise_( Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application flush_context.execute() Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 31 00:55:31.589514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rec.execute(self) Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 31 00:55:31.591002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application result = self._query(query) Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application conn.query(q) Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application result.read() Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 31 00:55:31.592611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [parameters: {'id': 'dd1a476d188d451283954eb3cfc89829', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:31.596041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:31.596814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:31.597865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:31.597865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Jul 31 00:55:31.597865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Jul 31 00:55:31.597865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:31.597865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:31.597865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Jul 31 00:55:31.597865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.Conflict( Jul 31 00:55:31.597865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id dd1a476d188d451283954eb3cfc89829. Jul 31 00:55:31.597865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:31.597865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 466/947] 10.222.0.42 () {60 vars in 1140 bytes} [Sun Jul 31 00:55:31 2022] POST /identity/v3/projects => generated 262 bytes in 275 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:31.598405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca01c65a-18bd-40e3-a523-0045129c10ca None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:31.649076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:31.737648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca01c65a-18bd-40e3-a523-0045129c10ca None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:31.775337 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33432fb-acd6-4650-8cb6-4267e7550786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:31.776012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33432fb-acd6-4650-8cb6-4267e7550786 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:31.776289 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33432fb-acd6-4650-8cb6-4267e7550786 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:31.776556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33432fb-acd6-4650-8cb6-4267e7550786 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:31.784302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33432fb-acd6-4650-8cb6-4267e7550786 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:31.785777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33432fb-acd6-4650-8cb6-4267e7550786 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-f33432fb-acd6-4650-8cb6-4267e7550786 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:31.798646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:31.799629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:31.800780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:31.800780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:31.800780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:31.800780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:31.800780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:31.800780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:31.800780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:31.800780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:31.800780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 482/948] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:31.808288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cae040-3822-4d97-ae92-51b372986e71 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:31.948456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cae040-3822-4d97-ae92-51b372986e71 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:31.990929 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56df57c8-48d7-4ff9-aa72-eb2f15f86819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:31.991586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56df57c8-48d7-4ff9-aa72-eb2f15f86819 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:31.991932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56df57c8-48d7-4ff9-aa72-eb2f15f86819 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:31.992282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56df57c8-48d7-4ff9-aa72-eb2f15f86819 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:31.993390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56df57c8-48d7-4ff9-aa72-eb2f15f86819 None admin] RBAC: Adding query filter params (name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:31.993796 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56df57c8-48d7-4ff9-aa72-eb2f15f86819 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:31.995662 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56df57c8-48d7-4ff9-aa72-eb2f15f86819 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:32.019419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 467/949] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:31 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:33.334039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2223cd5e-6d85-49d4-a563-0dc29c553ebf None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:33.334874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2223cd5e-6d85-49d4-a563-0dc29c553ebf None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:33.334874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2223cd5e-6d85-49d4-a563-0dc29c553ebf None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:33.336379 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 483/950] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:55:33.342274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d11c07-5ea1-48dd-84d5-5b81b2d41625 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:33.343138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d11c07-5ea1-48dd-84d5-5b81b2d41625 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:33.343138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d11c07-5ea1-48dd-84d5-5b81b2d41625 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:33.365871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a25797e-38a1-472d-82c5-d31c37321342 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:33.366159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a25797e-38a1-472d-82c5-d31c37321342 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:33.366460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a25797e-38a1-472d-82c5-d31c37321342 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:33.368163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 484/951] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:55:33.373499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66b91b3-d2b7-416b-b07d-f927dcd434a2 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:33.374528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66b91b3-d2b7-416b-b07d-f927dcd434a2 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:33.375002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66b91b3-d2b7-416b-b07d-f927dcd434a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:33.375247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b1d11c07-5ea1-48dd-84d5-5b81b2d41625 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:33.377723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b1d11c07-5ea1-48dd-84d5-5b81b2d41625 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:33.404549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a66b91b3-d2b7-416b-b07d-f927dcd434a2 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:33.406599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a66b91b3-d2b7-416b-b07d-f927dcd434a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:33.466074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:33.512198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 468/952] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:33 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:33.550894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 485/953] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:33 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:33.564843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d11c07-5ea1-48dd-84d5-5b81b2d41625 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:33.595545 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66b91b3-d2b7-416b-b07d-f927dcd434a2 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:33.648263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:33.770950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d11c07-5ea1-48dd-84d5-5b81b2d41625 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:33.812598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66b91b3-d2b7-416b-b07d-f927dcd434a2 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:33.840479 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d84471-58ac-4b09-b70a-529015da32b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:33.841921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d84471-58ac-4b09-b70a-529015da32b0 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:33.843293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d84471-58ac-4b09-b70a-529015da32b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:33.844014 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d84471-58ac-4b09-b70a-529015da32b0 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:33.845526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d84471-58ac-4b09-b70a-529015da32b0 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:33.848359 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d84471-58ac-4b09-b70a-529015da32b0 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-e5d84471-58ac-4b09-b70a-529015da32b0 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:33.867285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:33.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 31 00:55:33.871639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:33.876201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 469/954] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:33 2022] GET /identity/v3/services/compute => generated 88 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:33.885957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d84471-58ac-4b09-b70a-529015da32b0 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:33.890626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09873366-4408-4c96-9b04-70b2904afaca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:33.892256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09873366-4408-4c96-9b04-70b2904afaca None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:33.893140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09873366-4408-4c96-9b04-70b2904afaca None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:33.893865 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09873366-4408-4c96-9b04-70b2904afaca None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:33.915213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09873366-4408-4c96-9b04-70b2904afaca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-09873366-4408-4c96-9b04-70b2904afaca None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:33.926894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:33.928834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:33.930622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:33.930622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:33.930622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:33.930622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:33.930622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:33.930622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:33.930622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:33.930622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:33.930622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:33.930622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:33.930622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 31 00:55:33.930622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:33.943614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 486/955] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Jul 31 00:55:33 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 341 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:33.945541 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09873366-4408-4c96-9b04-70b2904afaca None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:33.955952 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:34.087236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d84471-58ac-4b09-b70a-529015da32b0 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:34.098096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:34.132138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09873366-4408-4c96-9b04-70b2904afaca None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:34.143952 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d3bae8-7833-49f0-8ea6-cfbffc093bbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:34.145314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d3bae8-7833-49f0-8ea6-cfbffc093bbe None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:34.145701 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d3bae8-7833-49f0-8ea6-cfbffc093bbe None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:34.146063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d3bae8-7833-49f0-8ea6-cfbffc093bbe None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:34.146881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d3bae8-7833-49f0-8ea6-cfbffc093bbe None admin] RBAC: Adding query filter params (name=compute) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:34.147273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d3bae8-7833-49f0-8ea6-cfbffc093bbe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:34.149294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d3bae8-7833-49f0-8ea6-cfbffc093bbe None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:34.159409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 470/956] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:33 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:34.160117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb813cc8-41fb-4019-88a1-6c5043b9ff45 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:34.160594 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb813cc8-41fb-4019-88a1-6c5043b9ff45 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:34.161070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb813cc8-41fb-4019-88a1-6c5043b9ff45 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:34.164226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 471/957] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:34.165984 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb813cc8-41fb-4019-88a1-6c5043b9ff45 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:34.198528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb24ab72-ee8b-4cb1-a66f-038102afe48b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:34.199460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb24ab72-ee8b-4cb1-a66f-038102afe48b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:34.199849 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb24ab72-ee8b-4cb1-a66f-038102afe48b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:34.200222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb24ab72-ee8b-4cb1-a66f-038102afe48b None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:34.201269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb24ab72-ee8b-4cb1-a66f-038102afe48b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:34.201671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb24ab72-ee8b-4cb1-a66f-038102afe48b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:34.203197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb24ab72-ee8b-4cb1-a66f-038102afe48b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:34.224501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 487/958] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Jul 31 00:55:33 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:34.226847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675874ca-2e0f-42e1-9fa7-08aab375dd7d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:34.227320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675874ca-2e0f-42e1-9fa7-08aab375dd7d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:34.227836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675874ca-2e0f-42e1-9fa7-08aab375dd7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:34.261155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-675874ca-2e0f-42e1-9fa7-08aab375dd7d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:34.263288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-675874ca-2e0f-42e1-9fa7-08aab375dd7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:34.319970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:34.326733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb813cc8-41fb-4019-88a1-6c5043b9ff45 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:34.371207 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2ca5af-8c4a-4138-98d3-d867f688585b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:34.372620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2ca5af-8c4a-4138-98d3-d867f688585b None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:34.373091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2ca5af-8c4a-4138-98d3-d867f688585b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:34.374188 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2ca5af-8c4a-4138-98d3-d867f688585b None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:34.375153 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2ca5af-8c4a-4138-98d3-d867f688585b None admin] RBAC: Adding query filter params (type=compute) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:34.375694 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2ca5af-8c4a-4138-98d3-d867f688585b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:34.378037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2ca5af-8c4a-4138-98d3-d867f688585b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:34.395475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 472/959] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:34 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:34.395475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2ca5af-8c4a-4138-98d3-d867f688585b None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:34.399050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 488/960] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:34 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:34.427931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675874ca-2e0f-42e1-9fa7-08aab375dd7d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:34.557867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:34.587884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2ca5af-8c4a-4138-98d3-d867f688585b None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:34.600820 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675874ca-2e0f-42e1-9fa7-08aab375dd7d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:34.636935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0217e939-5f98-4376-95c2-5a957d77bfe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:34.637952 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0217e939-5f98-4376-95c2-5a957d77bfe0 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:34.638434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0217e939-5f98-4376-95c2-5a957d77bfe0 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:34.638922 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0217e939-5f98-4376-95c2-5a957d77bfe0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:34.651689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0217e939-5f98-4376-95c2-5a957d77bfe0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:34.653518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0217e939-5f98-4376-95c2-5a957d77bfe0 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:34.667401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f733eef-2ae2-415d-8ef8-ea62854d483f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:34.668311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f733eef-2ae2-415d-8ef8-ea62854d483f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:34.668649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f733eef-2ae2-415d-8ef8-ea62854d483f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:34.669023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f733eef-2ae2-415d-8ef8-ea62854d483f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:34.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f733eef-2ae2-415d-8ef8-ea62854d483f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:34.672058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f733eef-2ae2-415d-8ef8-ea62854d483f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-0217e939-5f98-4376-95c2-5a957d77bfe0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:34.673317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:34.674297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:34.675729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:34.675729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:34.675729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:34.675729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:34.678034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 473/961] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:34.687641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0217e939-5f98-4376-95c2-5a957d77bfe0 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-6f733eef-2ae2-415d-8ef8-ea62854d483f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:34.691381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:34.692385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:34.693204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:34.694422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 489/962] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Jul 31 00:55:34 2022] GET /identity/v3/services/object-store => generated 93 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:34.704858 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f733eef-2ae2-415d-8ef8-ea62854d483f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:34.716239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:34.887187 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0217e939-5f98-4376-95c2-5a957d77bfe0 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:34.888332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f733eef-2ae2-415d-8ef8-ea62854d483f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:34.955522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a734b66-11d0-4b6d-a236-7bc2a9967426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:34.956819 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a734b66-11d0-4b6d-a236-7bc2a9967426 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:34.957487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a734b66-11d0-4b6d-a236-7bc2a9967426 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:34.958145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a734b66-11d0-4b6d-a236-7bc2a9967426 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:34.959634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a734b66-11d0-4b6d-a236-7bc2a9967426 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:34.960325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a734b66-11d0-4b6d-a236-7bc2a9967426 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:34.960833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09c1eec-546a-45fc-881b-0cdefb6e3a3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:34.961789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09c1eec-546a-45fc-881b-0cdefb6e3a3a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:34.962231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09c1eec-546a-45fc-881b-0cdefb6e3a3a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:34.962690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09c1eec-546a-45fc-881b-0cdefb6e3a3a None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:34.963829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a734b66-11d0-4b6d-a236-7bc2a9967426 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:34.964013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09c1eec-546a-45fc-881b-0cdefb6e3a3a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:34.964395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09c1eec-546a-45fc-881b-0cdefb6e3a3a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:34.967116 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09c1eec-546a-45fc-881b-0cdefb6e3a3a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:34.980029 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 490/963] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Jul 31 00:55:34 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:34.984622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a734b66-11d0-4b6d-a236-7bc2a9967426 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:34.992729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 474/964] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:34.997661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09c1eec-546a-45fc-881b-0cdefb6e3a3a None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:35.003119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:35.138012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:35.147019 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a734b66-11d0-4b6d-a236-7bc2a9967426 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:35.179098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09c1eec-546a-45fc-881b-0cdefb6e3a3a None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:35.210879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2adc0966-e5e0-4b95-942d-60dd8c425df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:35.211985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adc0966-e5e0-4b95-942d-60dd8c425df9 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:35.212491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adc0966-e5e0-4b95-942d-60dd8c425df9 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:35.212941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adc0966-e5e0-4b95-942d-60dd8c425df9 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:35.214077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2adc0966-e5e0-4b95-942d-60dd8c425df9 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:35.214571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2adc0966-e5e0-4b95-942d-60dd8c425df9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:35.217136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2adc0966-e5e0-4b95-942d-60dd8c425df9 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:35.235107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 491/965] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Jul 31 00:55:34 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:35.238555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2adc0966-e5e0-4b95-942d-60dd8c425df9 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:35.240382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff731bd-0cc0-4015-8835-e9657370e51d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:35.241371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff731bd-0cc0-4015-8835-e9657370e51d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:35.241806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff731bd-0cc0-4015-8835-e9657370e51d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:35.242245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff731bd-0cc0-4015-8835-e9657370e51d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:35.255151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff731bd-0cc0-4015-8835-e9657370e51d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:35.257719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff731bd-0cc0-4015-8835-e9657370e51d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-3ff731bd-0cc0-4015-8835-e9657370e51d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:35.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:35.281243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:35.282179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:35.282179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:35.282179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:35.282179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:35.282179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:35.282179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:35.282179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:35.282179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:35.282179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 31 00:55:35.282179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:35.288439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 475/966] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Jul 31 00:55:34 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:35.291903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff731bd-0cc0-4015-8835-e9657370e51d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:35.368629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:35.414952 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2adc0966-e5e0-4b95-942d-60dd8c425df9 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:35.459885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff731bd-0cc0-4015-8835-e9657370e51d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:35.471102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d12df8-8591-4d2a-a7c6-e9c103acfe67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:35.472493 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d12df8-8591-4d2a-a7c6-e9c103acfe67 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:35.472902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d12df8-8591-4d2a-a7c6-e9c103acfe67 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:35.473756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d12df8-8591-4d2a-a7c6-e9c103acfe67 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:35.475218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d12df8-8591-4d2a-a7c6-e9c103acfe67 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:35.477509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d12df8-8591-4d2a-a7c6-e9c103acfe67 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:35.503900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595de92f-9b8d-4cfb-9738-948cd25a9423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:35.504880 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595de92f-9b8d-4cfb-9738-948cd25a9423 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:35.505274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595de92f-9b8d-4cfb-9738-948cd25a9423 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:35.505720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595de92f-9b8d-4cfb-9738-948cd25a9423 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:35.507102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595de92f-9b8d-4cfb-9738-948cd25a9423 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:35.507650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595de92f-9b8d-4cfb-9738-948cd25a9423 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:35.510226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595de92f-9b8d-4cfb-9738-948cd25a9423 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:35.516215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 492/967] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 00:55:35 2022] POST /identity/v3/endpoints => generated 349 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 00:55:35.517882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:35.518704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa202de-7881-4f13-bfdd-cf13ca55ecdf None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:35.519846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa202de-7881-4f13-bfdd-cf13ca55ecdf None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:35.519846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa202de-7881-4f13-bfdd-cf13ca55ecdf None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:35.529667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 493/968] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:35 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:35.532129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb48ef4-a705-4277-856e-75e160dc8a16 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:35.532681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb48ef4-a705-4277-856e-75e160dc8a16 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:35.533262 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb48ef4-a705-4277-856e-75e160dc8a16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:35.567093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-9bb48ef4-a705-4277-856e-75e160dc8a16 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:35.569682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9bb48ef4-a705-4277-856e-75e160dc8a16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:35.580886 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 476/969] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Jul 31 00:55:35 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:35.586599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595de92f-9b8d-4cfb-9738-948cd25a9423 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:35.714236 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 494/970] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:35 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:35.754466 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb48ef4-a705-4277-856e-75e160dc8a16 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:35.784356 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595de92f-9b8d-4cfb-9738-948cd25a9423 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:35.842225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc75be1-6db3-41ca-9e7c-b6af9e3e89cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:35.845437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc75be1-6db3-41ca-9e7c-b6af9e3e89cc None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:35.845437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc75be1-6db3-41ca-9e7c-b6af9e3e89cc None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:35.845437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc75be1-6db3-41ca-9e7c-b6af9e3e89cc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:35.871433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:35.956757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:35.965303 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb48ef4-a705-4277-856e-75e160dc8a16 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:35.972597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc75be1-6db3-41ca-9e7c-b6af9e3e89cc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:35.975675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc75be1-6db3-41ca-9e7c-b6af9e3e89cc None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:36.026482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd2d836-c0a7-4420-84ee-45e00806387e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:36.027451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd2d836-c0a7-4420-84ee-45e00806387e None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:36.028050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd2d836-c0a7-4420-84ee-45e00806387e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:36.028511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd2d836-c0a7-4420-84ee-45e00806387e None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:36.029699 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd2d836-c0a7-4420-84ee-45e00806387e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:36.032283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd2d836-c0a7-4420-84ee-45e00806387e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:36.036724 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: WARNING keystone.server.flask.common [None req-9bd2d836-c0a7-4420-84ee-45e00806387e None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 31 00:55:36.058723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.sql.core [None req-9bd2d836-c0a7-4420-84ee-45e00806387e None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 31 00:55:36.058723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 31 00:55:36.058723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [parameters: {'user_id': 'a9d3560cf76646a49b0f1167b22a516b', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 31 00:55:36.058723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80209) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-9bd2d836-c0a7-4420-84ee-45e00806387e None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application result = self._query(query) Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application conn.query(q) Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application result.read() Jul 31 00:55:36.091870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 31 00:55:36.093956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application next(self.gen) Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application yield resource Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application next(self.gen) Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self.session.rollback() Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self.force_reraise() Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise self.value Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application session.commit() Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 31 00:55:36.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._prepare_impl() Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self.session.flush() Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._flush(objects) Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application compat.raise_( Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application flush_context.execute() Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rec.execute(self) Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 31 00:55:36.097963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application result = connection._execute_20( Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 31 00:55:36.099843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application result = self._query(query) Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application conn.query(q) Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application result.read() Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 31 00:55:36.103846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [parameters: {'user_id': 'a9d3560cf76646a49b0f1167b22a516b', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:36.106589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Jul 31 00:55:36.110586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Jul 31 00:55:36.115342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Jul 31 00:55:36.115342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Jul 31 00:55:36.115342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 31 00:55:36.115342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 31 00:55:36.115342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 31 00:55:36.115342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:36.115342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 495/971] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Jul 31 00:55:35 2022] POST /identity/v3/users => generated 160 bytes in 341 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:36.115342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd2d836-c0a7-4420-84ee-45e00806387e None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:36.187941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 477/972] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:35 2022] GET /identity/v3/auth/tokens => generated 1464 bytes in 602 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 31 00:55:36.195430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc75be1-6db3-41ca-9e7c-b6af9e3e89cc None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:36.268549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd2d836-c0a7-4420-84ee-45e00806387e None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:36.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:36.312791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590e9c99-7d2c-4892-aa07-22d3bb604dbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:36.314031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590e9c99-7d2c-4892-aa07-22d3bb604dbb None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:36.314522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590e9c99-7d2c-4892-aa07-22d3bb604dbb None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:36.316191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590e9c99-7d2c-4892-aa07-22d3bb604dbb None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:36.327637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590e9c99-7d2c-4892-aa07-22d3bb604dbb None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:36.329691 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590e9c99-7d2c-4892-aa07-22d3bb604dbb None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-590e9c99-7d2c-4892-aa07-22d3bb604dbb None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:36.343268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:36.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:36.345616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:36.345616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:36.345616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:36.345616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:36.345616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:36.345616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:36.345616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:36.345616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:36.345616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 31 00:55:36.345616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:36.348158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 496/973] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:55:36 2022] GET /identity/v3/users/cinder => generated 84 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:36.358594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590e9c99-7d2c-4892-aa07-22d3bb604dbb None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:36.378756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:36.394705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc75be1-6db3-41ca-9e7c-b6af9e3e89cc None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:36.456140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba523c8-ac31-48c0-ae27-c502fec140de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:36.457325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba523c8-ac31-48c0-ae27-c502fec140de None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:36.458324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba523c8-ac31-48c0-ae27-c502fec140de None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:36.458887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba523c8-ac31-48c0-ae27-c502fec140de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:36.478691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba523c8-ac31-48c0-ae27-c502fec140de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:36.481509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba523c8-ac31-48c0-ae27-c502fec140de None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-aba523c8-ac31-48c0-ae27-c502fec140de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:36.517070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:36.518246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:36.519559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:36.519559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:36.519559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:36.519559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:36.531715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 478/974] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:36.539920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba523c8-ac31-48c0-ae27-c502fec140de None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:36.560941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590e9c99-7d2c-4892-aa07-22d3bb604dbb None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:36.608994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e11e63c-c909-43c0-833d-10c6946503cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:36.609998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e11e63c-c909-43c0-833d-10c6946503cc None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:36.610738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e11e63c-c909-43c0-833d-10c6946503cc None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:36.611411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e11e63c-c909-43c0-833d-10c6946503cc None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:36.612519 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e11e63c-c909-43c0-833d-10c6946503cc None admin] RBAC: Adding query filter params (name=cinder) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:36.613296 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e11e63c-c909-43c0-833d-10c6946503cc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:36.615599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e11e63c-c909-43c0-833d-10c6946503cc None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:36.686142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 497/975] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Jul 31 00:55:36 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:36.731716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba523c8-ac31-48c0-ae27-c502fec140de None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:36.796039 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd463fdf-32f8-436e-a0be-48a1214a8bd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:36.798043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd463fdf-32f8-436e-a0be-48a1214a8bd6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:36.798808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd463fdf-32f8-436e-a0be-48a1214a8bd6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:36.799538 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd463fdf-32f8-436e-a0be-48a1214a8bd6 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:36.801369 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd463fdf-32f8-436e-a0be-48a1214a8bd6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:36.802058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd463fdf-32f8-436e-a0be-48a1214a8bd6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:36.803889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd463fdf-32f8-436e-a0be-48a1214a8bd6 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:36.818607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f9990b-740e-4c9c-8a6d-e8d1bd224fec None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:36.819199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f9990b-740e-4c9c-8a6d-e8d1bd224fec None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:36.819730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f9990b-740e-4c9c-8a6d-e8d1bd224fec None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:36.822014 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 498/976] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:36.823306 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 479/977] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:36.832065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f9990b-740e-4c9c-8a6d-e8d1bd224fec None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:36.832377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473780ef-0bcf-483d-8ea9-b04cb70eda26 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:36.832842 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473780ef-0bcf-483d-8ea9-b04cb70eda26 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:36.833410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473780ef-0bcf-483d-8ea9-b04cb70eda26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:36.847380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:36.854386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-473780ef-0bcf-483d-8ea9-b04cb70eda26 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:36.856362 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-473780ef-0bcf-483d-8ea9-b04cb70eda26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:36.900727 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:36.978463 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 480/978] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:36 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:37.008288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473780ef-0bcf-483d-8ea9-b04cb70eda26 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:37.026915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f9990b-740e-4c9c-8a6d-e8d1bd224fec None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:37.114621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f403a7-d949-48d3-ae91-542a7e7777e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:37.115738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f403a7-d949-48d3-ae91-542a7e7777e7 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:37.116236 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f403a7-d949-48d3-ae91-542a7e7777e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:37.116732 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f403a7-d949-48d3-ae91-542a7e7777e7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:37.130650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f403a7-d949-48d3-ae91-542a7e7777e7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:37.133968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f403a7-d949-48d3-ae91-542a7e7777e7 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-73f403a7-d949-48d3-ae91-542a7e7777e7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:37.157752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:37.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:37.169249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:37.169249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:37.169249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:37.169249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:37.169249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:37.169249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:37.169249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:37.169249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:37.169249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 499/979] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:37.170194 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f403a7-d949-48d3-ae91-542a7e7777e7 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:37.173838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473780ef-0bcf-483d-8ea9-b04cb70eda26 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:37.230194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f0a13-ba5c-4a6e-8492-9cd1d573e6be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:37.231697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f0a13-ba5c-4a6e-8492-9cd1d573e6be None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:37.232420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f0a13-ba5c-4a6e-8492-9cd1d573e6be None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:37.233211 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f0a13-ba5c-4a6e-8492-9cd1d573e6be None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:37.234498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316f0a13-ba5c-4a6e-8492-9cd1d573e6be None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:37.237060 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316f0a13-ba5c-4a6e-8492-9cd1d573e6be None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:37.244830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:37.267410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 481/980] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:55:37 2022] POST /identity/v3/services => generated 242 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:37.323971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:37.338377 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f403a7-d949-48d3-ae91-542a7e7777e7 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:37.393068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b51a38e-b718-434e-b871-905a0bd2e362 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:37.393829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b51a38e-b718-434e-b871-905a0bd2e362 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:37.394240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b51a38e-b718-434e-b871-905a0bd2e362 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:37.394767 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b51a38e-b718-434e-b871-905a0bd2e362 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:37.395856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b51a38e-b718-434e-b871-905a0bd2e362 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:37.396319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b51a38e-b718-434e-b871-905a0bd2e362 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:37.397962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b51a38e-b718-434e-b871-905a0bd2e362 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:37.418275 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 500/981] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:37.420771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f0a13-ba5c-4a6e-8492-9cd1d573e6be None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:37.615430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f0a13-ba5c-4a6e-8492-9cd1d573e6be None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:37.678309 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a64da2-db7d-41f1-b3e3-da8113478c83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:37.679226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a64da2-db7d-41f1-b3e3-da8113478c83 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:37.679720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a64da2-db7d-41f1-b3e3-da8113478c83 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:37.680168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a64da2-db7d-41f1-b3e3-da8113478c83 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:37.681556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a64da2-db7d-41f1-b3e3-da8113478c83 None admin] RBAC: Adding query filter params (role.id=53d929a69ffa476eac93c56d7de72a16, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=8cdffe91785e42d798b931082969598e) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:37.682089 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a64da2-db7d-41f1-b3e3-da8113478c83 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:37.684793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a64da2-db7d-41f1-b3e3-da8113478c83 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:37.696605 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 482/982] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:37 2022] GET /identity/v3/role_assignments?user.id=8cdffe91785e42d798b931082969598e&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=53d929a69ffa476eac93c56d7de72a16 => generated 608 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:37.899918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420c3977-26ae-431d-95d7-6a3e58949147 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:37.900732 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420c3977-26ae-431d-95d7-6a3e58949147 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:37.900732 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420c3977-26ae-431d-95d7-6a3e58949147 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:37.902109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 501/983] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:55:37.909251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdb4d70-97f3-4e72-9443-3e863765bab3 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:37.909875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdb4d70-97f3-4e72-9443-3e863765bab3 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:37.910411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdb4d70-97f3-4e72-9443-3e863765bab3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:37.940943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-8bdb4d70-97f3-4e72-9443-3e863765bab3 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:37.942831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8bdb4d70-97f3-4e72-9443-3e863765bab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:38.028782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:38.045249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 483/984] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:37 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:38.068976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420c3977-26ae-431d-95d7-6a3e58949147 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:38.253331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420c3977-26ae-431d-95d7-6a3e58949147 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:38.320943 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d691188-cb0f-4e79-985e-560b2e4ca9d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:38.321957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d691188-cb0f-4e79-985e-560b2e4ca9d3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:38.322396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d691188-cb0f-4e79-985e-560b2e4ca9d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:38.322829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d691188-cb0f-4e79-985e-560b2e4ca9d3 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:38.323923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d691188-cb0f-4e79-985e-560b2e4ca9d3 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:38.326248 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d691188-cb0f-4e79-985e-560b2e4ca9d3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-3d691188-cb0f-4e79-985e-560b2e4ca9d3 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:38.344801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:38.346564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:38.348665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 502/985] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Jul 31 00:55:38 2022] GET /identity/v3/services/object-store => generated 93 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:38.357034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdb4d70-97f3-4e72-9443-3e863765bab3 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:38.478147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdb4d70-97f3-4e72-9443-3e863765bab3 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:38.519961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90881a7-75d2-4f58-b29b-c7e58610af96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:38.520541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90881a7-75d2-4f58-b29b-c7e58610af96 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:38.523680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90881a7-75d2-4f58-b29b-c7e58610af96 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:38.523680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90881a7-75d2-4f58-b29b-c7e58610af96 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:38.523680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90881a7-75d2-4f58-b29b-c7e58610af96 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:38.523680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90881a7-75d2-4f58-b29b-c7e58610af96 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:38.523680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90881a7-75d2-4f58-b29b-c7e58610af96 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:38.534420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 484/986] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Jul 31 00:55:38 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:38.540020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d691188-cb0f-4e79-985e-560b2e4ca9d3 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:38.559577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:38.677902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d691188-cb0f-4e79-985e-560b2e4ca9d3 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:38.723658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae44002a-1c0c-463f-9d08-5284ef1f24f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:38.724417 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae44002a-1c0c-463f-9d08-5284ef1f24f5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:38.724699 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae44002a-1c0c-463f-9d08-5284ef1f24f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:38.724971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae44002a-1c0c-463f-9d08-5284ef1f24f5 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:38.725801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae44002a-1c0c-463f-9d08-5284ef1f24f5 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:38.726146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae44002a-1c0c-463f-9d08-5284ef1f24f5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:38.728093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae44002a-1c0c-463f-9d08-5284ef1f24f5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:38.735474 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 503/987] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Jul 31 00:55:38 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:38.742174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90881a7-75d2-4f58-b29b-c7e58610af96 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:38.809699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:38.859602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90881a7-75d2-4f58-b29b-c7e58610af96 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:38.894248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b471c6e9-cfa0-4a52-a7f7-72256cc5988c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:38.894868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b471c6e9-cfa0-4a52-a7f7-72256cc5988c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:38.895146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b471c6e9-cfa0-4a52-a7f7-72256cc5988c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:38.895449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b471c6e9-cfa0-4a52-a7f7-72256cc5988c None admin] PATH_INFO: `/v3/endpoints` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:38.896209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b471c6e9-cfa0-4a52-a7f7-72256cc5988c None admin] RBAC: Adding query filter params (interface=admin, service_id=975b02a2f0824e02bf7ba610a874756e, region_id=RegionOne) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:38.896515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b471c6e9-cfa0-4a52-a7f7-72256cc5988c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:38.897882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b471c6e9-cfa0-4a52-a7f7-72256cc5988c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:38.909870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 485/988] 10.222.0.42 () {56 vars in 1249 bytes} [Sun Jul 31 00:55:38 2022] GET /identity/v3/endpoints?service_id=975b02a2f0824e02bf7ba610a874756e&interface=admin®ion_id=RegionOne => generated 193 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:38.911715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae44002a-1c0c-463f-9d08-5284ef1f24f5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:39.005639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:39.034900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae44002a-1c0c-463f-9d08-5284ef1f24f5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:39.082273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c35c080-0251-4495-9455-ce75b99747d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:39.083167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c35c080-0251-4495-9455-ce75b99747d4 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:39.083657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c35c080-0251-4495-9455-ce75b99747d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:39.084113 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c35c080-0251-4495-9455-ce75b99747d4 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:39.085276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c35c080-0251-4495-9455-ce75b99747d4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:39.087757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c35c080-0251-4495-9455-ce75b99747d4 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:39.098155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 504/989] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Jul 31 00:55:38 2022] GET /identity/v3/services => generated 1727 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 00:55:39.894446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893e6897-7078-41dc-bc1f-e4a59ed58a0d None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:39.894962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893e6897-7078-41dc-bc1f-e4a59ed58a0d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:39.894962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893e6897-7078-41dc-bc1f-e4a59ed58a0d None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:39.896489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 486/990] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:39.901642 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6acfa3-76a9-4c0f-bf11-6717da4b6f3d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:39.901925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6acfa3-76a9-4c0f-bf11-6717da4b6f3d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:39.902193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6acfa3-76a9-4c0f-bf11-6717da4b6f3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:39.920848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-0d6acfa3-76a9-4c0f-bf11-6717da4b6f3d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:39.922566 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0d6acfa3-76a9-4c0f-bf11-6717da4b6f3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:40.001942 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 505/991] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:39 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:40.023721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893e6897-7078-41dc-bc1f-e4a59ed58a0d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:40.123073 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a33797-59dd-453d-9a2b-b01533650098 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:40.124061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a33797-59dd-453d-9a2b-b01533650098 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:40.124241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a33797-59dd-453d-9a2b-b01533650098 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:40.125798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 506/992] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:55:40.132005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3777ca2-f0dc-4216-87b9-eb5c7380ace4 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:40.132298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3777ca2-f0dc-4216-87b9-eb5c7380ace4 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:40.132561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3777ca2-f0dc-4216-87b9-eb5c7380ace4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:40.148218 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893e6897-7078-41dc-bc1f-e4a59ed58a0d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:40.151800 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-e3777ca2-f0dc-4216-87b9-eb5c7380ace4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:40.153038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e3777ca2-f0dc-4216-87b9-eb5c7380ace4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:40.169218 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:40.189911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d736241-df76-4c19-953c-7f9060f87eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:40.190623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d736241-df76-4c19-953c-7f9060f87eb4 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:40.190845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d736241-df76-4c19-953c-7f9060f87eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:40.191107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d736241-df76-4c19-953c-7f9060f87eb4 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:40.191773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d736241-df76-4c19-953c-7f9060f87eb4 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:40.193170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d736241-df76-4c19-953c-7f9060f87eb4 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-7d736241-df76-4c19-953c-7f9060f87eb4 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:40.203755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:40.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:40.206175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 487/993] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:40 2022] GET /identity/v3/services/compute => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:40.211011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d736241-df76-4c19-953c-7f9060f87eb4 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:40.233739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:40.240231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 507/994] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:40 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:40.262316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3777ca2-f0dc-4216-87b9-eb5c7380ace4 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:40.336513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d736241-df76-4c19-953c-7f9060f87eb4 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:40.373316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3777ca2-f0dc-4216-87b9-eb5c7380ace4 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:40.383878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05083d18-cdca-4c10-8017-54d03929105e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:40.384605 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05083d18-cdca-4c10-8017-54d03929105e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:40.384805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05083d18-cdca-4c10-8017-54d03929105e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:40.385054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05083d18-cdca-4c10-8017-54d03929105e None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:40.385732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05083d18-cdca-4c10-8017-54d03929105e None admin] RBAC: Adding query filter params (name=compute) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:40.386035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05083d18-cdca-4c10-8017-54d03929105e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:40.387446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05083d18-cdca-4c10-8017-54d03929105e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:40.397243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 488/995] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:40 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:40.398301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05083d18-cdca-4c10-8017-54d03929105e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:40.412455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed22bce-1478-4b25-af73-a52a197e587d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:40.413107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed22bce-1478-4b25-af73-a52a197e587d None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:40.413383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed22bce-1478-4b25-af73-a52a197e587d None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:40.413652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed22bce-1478-4b25-af73-a52a197e587d None admin] PATH_INFO: `/v3/services/image` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:40.414273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed22bce-1478-4b25-af73-a52a197e587d None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:40.415681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed22bce-1478-4b25-af73-a52a197e587d None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-1ed22bce-1478-4b25-af73-a52a197e587d None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:40.425259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:40.426513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:40.427397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 508/996] 10.222.0.42 () {56 vars in 1105 bytes} [Sun Jul 31 00:55:40 2022] GET /identity/v3/services/image => generated 86 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:40.434004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed22bce-1478-4b25-af73-a52a197e587d None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:40.457863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:40.498216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:40.517651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05083d18-cdca-4c10-8017-54d03929105e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:40.546803 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed22bce-1478-4b25-af73-a52a197e587d None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:40.561438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718ed178-3503-4e1a-a49c-ac176eb52237 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:40.562054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718ed178-3503-4e1a-a49c-ac176eb52237 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:40.562325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718ed178-3503-4e1a-a49c-ac176eb52237 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:40.562592 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718ed178-3503-4e1a-a49c-ac176eb52237 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:40.563255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718ed178-3503-4e1a-a49c-ac176eb52237 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:40.563582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718ed178-3503-4e1a-a49c-ac176eb52237 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:40.564935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718ed178-3503-4e1a-a49c-ac176eb52237 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:40.570230 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 489/997] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:40 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:40.575949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718ed178-3503-4e1a-a49c-ac176eb52237 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:40.585698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7887cc-9b12-4b29-a9a7-8d09ac8d9452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:40.586318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7887cc-9b12-4b29-a9a7-8d09ac8d9452 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:40.586603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7887cc-9b12-4b29-a9a7-8d09ac8d9452 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:40.586873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7887cc-9b12-4b29-a9a7-8d09ac8d9452 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:40.587568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7887cc-9b12-4b29-a9a7-8d09ac8d9452 None admin] RBAC: Adding query filter params (name=image) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:40.587871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7887cc-9b12-4b29-a9a7-8d09ac8d9452 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:40.589226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7887cc-9b12-4b29-a9a7-8d09ac8d9452 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:40.595038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 509/998] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:40 2022] GET /identity/v3/services?name=image => generated 122 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:40.600498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7887cc-9b12-4b29-a9a7-8d09ac8d9452 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:40.686112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718ed178-3503-4e1a-a49c-ac176eb52237 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:40.699092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:40.712903 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7887cc-9b12-4b29-a9a7-8d09ac8d9452 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:40.728067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57abac1-d79d-4bb6-b9e3-fd8403ff4d1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:40.728771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57abac1-d79d-4bb6-b9e3-fd8403ff4d1d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:40.729051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57abac1-d79d-4bb6-b9e3-fd8403ff4d1d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:40.729331 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57abac1-d79d-4bb6-b9e3-fd8403ff4d1d None admin] PATH_INFO: `/v3/endpoints` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:40.730053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57abac1-d79d-4bb6-b9e3-fd8403ff4d1d None admin] RBAC: Adding query filter params (interface=public, service_id=b480ee014a5241adb262e2ceac3e3f4f, region_id=RegionOne) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:40.730360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57abac1-d79d-4bb6-b9e3-fd8403ff4d1d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:40.731748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57abac1-d79d-4bb6-b9e3-fd8403ff4d1d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:40.737677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 490/999] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Jul 31 00:55:40 2022] GET /identity/v3/endpoints?service_id=b480ee014a5241adb262e2ceac3e3f4f&interface=public®ion_id=RegionOne => generated 194 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:40.741769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:40.743138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57abac1-d79d-4bb6-b9e3-fd8403ff4d1d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:40.751547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b01b6a-a44a-4aa1-8a1e-38aea71af8b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:40.752172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b01b6a-a44a-4aa1-8a1e-38aea71af8b4 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:40.752456 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b01b6a-a44a-4aa1-8a1e-38aea71af8b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:40.752730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b01b6a-a44a-4aa1-8a1e-38aea71af8b4 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:40.753415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b01b6a-a44a-4aa1-8a1e-38aea71af8b4 None admin] RBAC: Adding query filter params (type=image) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:40.753720 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b01b6a-a44a-4aa1-8a1e-38aea71af8b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:40.755079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b01b6a-a44a-4aa1-8a1e-38aea71af8b4 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:40.767384 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 510/1000] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:40 2022] GET /identity/v3/services?type=image => generated 122 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:40.863015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57abac1-d79d-4bb6-b9e3-fd8403ff4d1d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:40.899706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe17a207-cb70-4c44-94d9-647959424df7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:40.900338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe17a207-cb70-4c44-94d9-647959424df7 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:40.900595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe17a207-cb70-4c44-94d9-647959424df7 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:40.900869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe17a207-cb70-4c44-94d9-647959424df7 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:40.901528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe17a207-cb70-4c44-94d9-647959424df7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:40.902912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe17a207-cb70-4c44-94d9-647959424df7 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:40.915379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 491/1001] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Jul 31 00:55:40 2022] GET /identity/v3/services => generated 1727 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 00:55:41.271371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc8c450-8a33-4347-b349-8dbc2d5272f1 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:41.271902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc8c450-8a33-4347-b349-8dbc2d5272f1 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:41.271978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc8c450-8a33-4347-b349-8dbc2d5272f1 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:41.273467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 511/1002] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:41.278759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492b4a31-39df-455c-b3ad-64c07ed3ff33 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:41.279053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492b4a31-39df-455c-b3ad-64c07ed3ff33 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:41.279331 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492b4a31-39df-455c-b3ad-64c07ed3ff33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:41.297423 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-492b4a31-39df-455c-b3ad-64c07ed3ff33 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:41.298604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-492b4a31-39df-455c-b3ad-64c07ed3ff33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:41.330096 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:41.369405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 492/1003] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:41 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:41.392646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc8c450-8a33-4347-b349-8dbc2d5272f1 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:41.505565 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc8c450-8a33-4347-b349-8dbc2d5272f1 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:41.558149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c943148-1ab9-446f-93e6-15b605272318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:41.558828 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c943148-1ab9-446f-93e6-15b605272318 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:41.559109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c943148-1ab9-446f-93e6-15b605272318 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:41.559407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c943148-1ab9-446f-93e6-15b605272318 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:41.560043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c943148-1ab9-446f-93e6-15b605272318 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:41.562084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c943148-1ab9-446f-93e6-15b605272318 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-9c943148-1ab9-446f-93e6-15b605272318 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:41.571686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:41.572778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 31 00:55:41.573603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:41.575015 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 512/1004] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Jul 31 00:55:41 2022] GET /identity/v3/services/object-store => generated 93 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:41.579048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492b4a31-39df-455c-b3ad-64c07ed3ff33 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:41.685218 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492b4a31-39df-455c-b3ad-64c07ed3ff33 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:41.720248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995410d9-b92b-4f7b-8441-569b19dc42eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:41.720866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995410d9-b92b-4f7b-8441-569b19dc42eb None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:41.721146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995410d9-b92b-4f7b-8441-569b19dc42eb None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:41.721421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995410d9-b92b-4f7b-8441-569b19dc42eb None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:41.722086 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995410d9-b92b-4f7b-8441-569b19dc42eb None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:41.722388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995410d9-b92b-4f7b-8441-569b19dc42eb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:41.723784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995410d9-b92b-4f7b-8441-569b19dc42eb None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:41.734116 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 493/1005] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Jul 31 00:55:41 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:41.735134 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c943148-1ab9-446f-93e6-15b605272318 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:41.793435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:41.842545 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c943148-1ab9-446f-93e6-15b605272318 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:41.880458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaea517-22f7-46e6-b057-012a756820a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:41.881087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaea517-22f7-46e6-b057-012a756820a2 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:41.881365 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaea517-22f7-46e6-b057-012a756820a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:41.881636 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaea517-22f7-46e6-b057-012a756820a2 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:41.882320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaea517-22f7-46e6-b057-012a756820a2 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:41.882620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaea517-22f7-46e6-b057-012a756820a2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:41.884029 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaea517-22f7-46e6-b057-012a756820a2 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:41.889831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 513/1006] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Jul 31 00:55:41 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:41.895932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995410d9-b92b-4f7b-8441-569b19dc42eb None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:41.930674 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:42.003204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995410d9-b92b-4f7b-8441-569b19dc42eb None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:42.051402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2416e495-a6c7-4ab7-87f4-f8894af2581e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:42.052029 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2416e495-a6c7-4ab7-87f4-f8894af2581e None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:42.052308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2416e495-a6c7-4ab7-87f4-f8894af2581e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:42.052604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2416e495-a6c7-4ab7-87f4-f8894af2581e None admin] PATH_INFO: `/v3/endpoints` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:42.053324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2416e495-a6c7-4ab7-87f4-f8894af2581e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:42.054685 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2416e495-a6c7-4ab7-87f4-f8894af2581e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:42.079406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 494/1007] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 00:55:41 2022] POST /identity/v3/endpoints => generated 325 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:42.796927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5408f5b2-d1e0-40ad-a358-c8d90eb61a35 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:42.797470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5408f5b2-d1e0-40ad-a358-c8d90eb61a35 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:42.797470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5408f5b2-d1e0-40ad-a358-c8d90eb61a35 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:42.799090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 514/1008] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:42.804878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09451e9d-54ad-4def-863d-d584f51e0f7d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:42.805131 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09451e9d-54ad-4def-863d-d584f51e0f7d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:42.805359 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09451e9d-54ad-4def-863d-d584f51e0f7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:42.824878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-09451e9d-54ad-4def-863d-d584f51e0f7d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:42.826085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-09451e9d-54ad-4def-863d-d584f51e0f7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:42.911470 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 495/1009] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:42 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:42.936058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e086604-1f2d-42dc-8bdd-841abaa4ce07 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:42.936264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e086604-1f2d-42dc-8bdd-841abaa4ce07 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:42.936588 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e086604-1f2d-42dc-8bdd-841abaa4ce07 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:42.938205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09451e9d-54ad-4def-863d-d584f51e0f7d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:42.938662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 515/1010] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:42.943636 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd7c17f-0950-490e-8300-ac6d91194eb6 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:42.943976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd7c17f-0950-490e-8300-ac6d91194eb6 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:42.944253 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd7c17f-0950-490e-8300-ac6d91194eb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:42.958412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:42.968672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-fdd7c17f-0950-490e-8300-ac6d91194eb6 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:42.969880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-fdd7c17f-0950-490e-8300-ac6d91194eb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:43.048839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 516/1011] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:42 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:43.051481 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09451e9d-54ad-4def-863d-d584f51e0f7d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:43.071416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd7c17f-0950-490e-8300-ac6d91194eb6 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:43.090759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05281eb7-66ce-4256-94a3-b2a45628aa6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:43.091658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05281eb7-66ce-4256-94a3-b2a45628aa6f None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:43.092607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05281eb7-66ce-4256-94a3-b2a45628aa6f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:43.092889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05281eb7-66ce-4256-94a3-b2a45628aa6f None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:43.093839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05281eb7-66ce-4256-94a3-b2a45628aa6f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:43.095904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05281eb7-66ce-4256-94a3-b2a45628aa6f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:43.114237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 496/1012] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:55:42 2022] POST /identity/v3/services => generated 242 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 00:55:43.142452 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:43.204015 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd7c17f-0950-490e-8300-ac6d91194eb6 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:43.278782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4eacb23-c2e0-46c2-aaeb-1c1cfe1f35df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:43.280054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eacb23-c2e0-46c2-aaeb-1c1cfe1f35df None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:43.280347 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eacb23-c2e0-46c2-aaeb-1c1cfe1f35df None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:43.280627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eacb23-c2e0-46c2-aaeb-1c1cfe1f35df None admin] PATH_INFO: `/v3/services/compute` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:43.281280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4eacb23-c2e0-46c2-aaeb-1c1cfe1f35df None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:43.282738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4eacb23-c2e0-46c2-aaeb-1c1cfe1f35df None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-d4eacb23-c2e0-46c2-aaeb-1c1cfe1f35df None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:43.293232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:43.295444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:43.296386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 517/1013] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:43 2022] GET /identity/v3/services/compute => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:43.301571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05281eb7-66ce-4256-94a3-b2a45628aa6f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:43.409846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:43.443431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05281eb7-66ce-4256-94a3-b2a45628aa6f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:43.484306 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f8f0fa-8377-4781-a9c8-310ef5e3c8a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:43.485311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f8f0fa-8377-4781-a9c8-310ef5e3c8a7 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:43.485779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f8f0fa-8377-4781-a9c8-310ef5e3c8a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:43.486249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f8f0fa-8377-4781-a9c8-310ef5e3c8a7 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:43.487403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f8f0fa-8377-4781-a9c8-310ef5e3c8a7 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:43.487924 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f8f0fa-8377-4781-a9c8-310ef5e3c8a7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:43.489987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f8f0fa-8377-4781-a9c8-310ef5e3c8a7 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:43.497590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 497/1014] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:43 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:43.504047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4eacb23-c2e0-46c2-aaeb-1c1cfe1f35df None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:43.627739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4eacb23-c2e0-46c2-aaeb-1c1cfe1f35df None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:43.639472 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:43.665444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4e3a49-8fc9-44d6-b98b-381ca106309b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:43.665981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e3a49-8fc9-44d6-b98b-381ca106309b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:43.666268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e3a49-8fc9-44d6-b98b-381ca106309b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:43.666526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e3a49-8fc9-44d6-b98b-381ca106309b None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:43.667439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e3a49-8fc9-44d6-b98b-381ca106309b None admin] RBAC: Adding query filter params (type=compute) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:43.667548 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e3a49-8fc9-44d6-b98b-381ca106309b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:43.668870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e3a49-8fc9-44d6-b98b-381ca106309b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:43.678857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 518/1015] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Jul 31 00:55:43 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:43.680101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f8f0fa-8377-4781-a9c8-310ef5e3c8a7 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:43.782086 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f8f0fa-8377-4781-a9c8-310ef5e3c8a7 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:43.816984 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02b1caa-f0c9-4cf8-a5a7-efff75b95fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:43.817593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02b1caa-f0c9-4cf8-a5a7-efff75b95fd5 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:43.817986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02b1caa-f0c9-4cf8-a5a7-efff75b95fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:43.818262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02b1caa-f0c9-4cf8-a5a7-efff75b95fd5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:43.818969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02b1caa-f0c9-4cf8-a5a7-efff75b95fd5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:43.820354 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02b1caa-f0c9-4cf8-a5a7-efff75b95fd5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:43.836595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:43.851386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 498/1016] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 00:55:43 2022] POST /identity/v3/endpoints => generated 334 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:44.142522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e85b58-5e00-42bd-9c2b-4799744742b3 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:44.143034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e85b58-5e00-42bd-9c2b-4799744742b3 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:44.143034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e85b58-5e00-42bd-9c2b-4799744742b3 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:44.144523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 519/1017] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:44.149619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec1fd21-7e48-4de8-bb44-fb4d9e836dcc None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:44.150040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec1fd21-7e48-4de8-bb44-fb4d9e836dcc None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:44.150205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec1fd21-7e48-4de8-bb44-fb4d9e836dcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:44.168565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-7ec1fd21-7e48-4de8-bb44-fb4d9e836dcc None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:44.169731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7ec1fd21-7e48-4de8-bb44-fb4d9e836dcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:44.243668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 499/1018] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:44 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:44.266150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e85b58-5e00-42bd-9c2b-4799744742b3 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:44.376005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e85b58-5e00-42bd-9c2b-4799744742b3 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:44.421369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413467a9-976e-49a4-9d3c-7cc144ee64dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:44.422008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413467a9-976e-49a4-9d3c-7cc144ee64dd None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:44.422307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413467a9-976e-49a4-9d3c-7cc144ee64dd None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:44.422576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413467a9-976e-49a4-9d3c-7cc144ee64dd None admin] PATH_INFO: `/v3/domains/default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:44.423739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413467a9-976e-49a4-9d3c-7cc144ee64dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:44.425160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413467a9-976e-49a4-9d3c-7cc144ee64dd None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:44.427179 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 520/1019] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:44.432909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec1fd21-7e48-4de8-bb44-fb4d9e836dcc None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:44.539945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec1fd21-7e48-4de8-bb44-fb4d9e836dcc None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:44.565435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:44.577816 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cefeb18-c567-4e39-a5b5-39c49fa319b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:44.578433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cefeb18-c567-4e39-a5b5-39c49fa319b0 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:44.578729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cefeb18-c567-4e39-a5b5-39c49fa319b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:44.579005 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cefeb18-c567-4e39-a5b5-39c49fa319b0 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:44.579716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cefeb18-c567-4e39-a5b5-39c49fa319b0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:44.581147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cefeb18-c567-4e39-a5b5-39c49fa319b0 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:44.615389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 500/1020] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:55:44 2022] POST /identity/v3/projects => generated 311 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:45.230210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f9b058-57c5-4def-8453-750cc1a87cd3 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:45.230738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f9b058-57c5-4def-8453-750cc1a87cd3 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:45.230738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f9b058-57c5-4def-8453-750cc1a87cd3 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:45.232112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 521/1021] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:45.237688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260ec0eb-6edc-4ea8-b00a-1f066529bb82 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:45.238106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260ec0eb-6edc-4ea8-b00a-1f066529bb82 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:45.238239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260ec0eb-6edc-4ea8-b00a-1f066529bb82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:45.257447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-260ec0eb-6edc-4ea8-b00a-1f066529bb82 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:45.258626 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-260ec0eb-6edc-4ea8-b00a-1f066529bb82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:45.327422 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 501/1022] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:45 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:45.349142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f9b058-57c5-4def-8453-750cc1a87cd3 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:45.398339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:45.463081 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f9b058-57c5-4def-8453-750cc1a87cd3 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:45.502212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a73234-4eec-435f-b176-015c10f21501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:45.502759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a73234-4eec-435f-b176-015c10f21501 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:45.503038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a73234-4eec-435f-b176-015c10f21501 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:45.503310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a73234-4eec-435f-b176-015c10f21501 None admin] PATH_INFO: `/v3/services/image` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:45.503961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a73234-4eec-435f-b176-015c10f21501 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:45.505329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a73234-4eec-435f-b176-015c10f21501 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-67a73234-4eec-435f-b176-015c10f21501 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:45.514756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:45.516203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:45.517030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 522/1023] 10.222.0.42 () {56 vars in 1105 bytes} [Sun Jul 31 00:55:45 2022] GET /identity/v3/services/image => generated 86 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:45.524679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260ec0eb-6edc-4ea8-b00a-1f066529bb82 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:45.640368 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260ec0eb-6edc-4ea8-b00a-1f066529bb82 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:45.648237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:45.682131 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65181d76-a17c-4231-a6ec-2622b93e4ab3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:45.683000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65181d76-a17c-4231-a6ec-2622b93e4ab3 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:45.683000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65181d76-a17c-4231-a6ec-2622b93e4ab3 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:45.683283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65181d76-a17c-4231-a6ec-2622b93e4ab3 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:45.683987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65181d76-a17c-4231-a6ec-2622b93e4ab3 None admin] RBAC: Adding query filter params (name=image) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:45.684416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65181d76-a17c-4231-a6ec-2622b93e4ab3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:45.685692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65181d76-a17c-4231-a6ec-2622b93e4ab3 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:45.691539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 502/1024] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:45 2022] GET /identity/v3/services?name=image => generated 122 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:45.696874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a73234-4eec-435f-b176-015c10f21501 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:45.853969 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a73234-4eec-435f-b176-015c10f21501 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:45.868057 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:45.914926 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe191f3-21cd-419d-a747-cb265cf08d75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:45.916044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe191f3-21cd-419d-a747-cb265cf08d75 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:45.916496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe191f3-21cd-419d-a747-cb265cf08d75 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:45.916999 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe191f3-21cd-419d-a747-cb265cf08d75 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:45.918675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe191f3-21cd-419d-a747-cb265cf08d75 None admin] RBAC: Adding query filter params (type=image) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:45.919211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe191f3-21cd-419d-a747-cb265cf08d75 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:45.922465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe191f3-21cd-419d-a747-cb265cf08d75 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:45.934444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 523/1025] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:45 2022] GET /identity/v3/services?type=image => generated 350 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:45.941195 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65181d76-a17c-4231-a6ec-2622b93e4ab3 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:45.979470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40901d82-4ece-4e40-852e-c2e20e100168 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:45.979737 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40901d82-4ece-4e40-852e-c2e20e100168 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:45.980024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40901d82-4ece-4e40-852e-c2e20e100168 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:45.981384 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 524/1026] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:55:45.986737 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b3af72-bded-4872-a2c6-a79f619967f8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:45.987020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b3af72-bded-4872-a2c6-a79f619967f8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:45.987295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b3af72-bded-4872-a2c6-a79f619967f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:46.005636 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-50b3af72-bded-4872-a2c6-a79f619967f8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:46.006919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-50b3af72-bded-4872-a2c6-a79f619967f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:46.069117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65181d76-a17c-4231-a6ec-2622b93e4ab3 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:46.084448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 525/1027] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:45 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:46.107892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1318732-57a1-4c38-966c-aab7a8a0757d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:46.108229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b3af72-bded-4872-a2c6-a79f619967f8 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:46.108425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1318732-57a1-4c38-966c-aab7a8a0757d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:46.108696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1318732-57a1-4c38-966c-aab7a8a0757d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:46.108963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1318732-57a1-4c38-966c-aab7a8a0757d None admin] PATH_INFO: `/v3/endpoints` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:46.109709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1318732-57a1-4c38-966c-aab7a8a0757d None admin] RBAC: Adding query filter params (interface=public, service_id=7aebb03cd6404d368f8436faebc4f4e1, region_id=RegionOne) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:46.110013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1318732-57a1-4c38-966c-aab7a8a0757d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:46.111400 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1318732-57a1-4c38-966c-aab7a8a0757d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:46.121994 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 503/1028] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Jul 31 00:55:45 2022] GET /identity/v3/endpoints?service_id=7aebb03cd6404d368f8436faebc4f4e1&interface=public®ion_id=RegionOne => generated 194 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:46.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1318732-57a1-4c38-966c-aab7a8a0757d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:46.168234 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:46.218213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:46.222344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b3af72-bded-4872-a2c6-a79f619967f8 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:46.251940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1318732-57a1-4c38-966c-aab7a8a0757d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:46.266483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3333b313-64a1-4c9f-b62f-43a82a0a9213 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:46.267129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333b313-64a1-4c9f-b62f-43a82a0a9213 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:46.267430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333b313-64a1-4c9f-b62f-43a82a0a9213 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:46.267709 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333b313-64a1-4c9f-b62f-43a82a0a9213 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:46.278961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3333b313-64a1-4c9f-b62f-43a82a0a9213 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-3333b313-64a1-4c9f-b62f-43a82a0a9213 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:46.284571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:46.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:46.286398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 526/1029] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Jul 31 00:55:46 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:46.291919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3333b313-64a1-4c9f-b62f-43a82a0a9213 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:46.295557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33758145-8d31-4639-a5ce-fe0deb1f7bb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:46.297120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33758145-8d31-4639-a5ce-fe0deb1f7bb9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:46.297411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33758145-8d31-4639-a5ce-fe0deb1f7bb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:46.297692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33758145-8d31-4639-a5ce-fe0deb1f7bb9 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:46.298352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33758145-8d31-4639-a5ce-fe0deb1f7bb9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:46.299778 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33758145-8d31-4639-a5ce-fe0deb1f7bb9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:46.307430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 504/1030] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Jul 31 00:55:46 2022] GET /identity/v3/services => generated 1957 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 00:55:46.411903 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3333b313-64a1-4c9f-b62f-43a82a0a9213 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:46.456068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4516998a-a96f-452e-bdef-68083e3505cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:46.456771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4516998a-a96f-452e-bdef-68083e3505cd None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:46.457050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4516998a-a96f-452e-bdef-68083e3505cd None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:46.457318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4516998a-a96f-452e-bdef-68083e3505cd None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:46.458058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4516998a-a96f-452e-bdef-68083e3505cd None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:46.458362 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4516998a-a96f-452e-bdef-68083e3505cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:46.459814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4516998a-a96f-452e-bdef-68083e3505cd None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:46.469957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 527/1031] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Jul 31 00:55:46 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:46.475592 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33758145-8d31-4639-a5ce-fe0deb1f7bb9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:46.616779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:46.622194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33758145-8d31-4639-a5ce-fe0deb1f7bb9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:46.666827 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2afe26f-dd71-47a3-824b-82178c79b8fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:46.667516 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2afe26f-dd71-47a3-824b-82178c79b8fe None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:46.667835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2afe26f-dd71-47a3-824b-82178c79b8fe None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:46.668274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2afe26f-dd71-47a3-824b-82178c79b8fe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:46.681903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2afe26f-dd71-47a3-824b-82178c79b8fe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:46.683426 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2afe26f-dd71-47a3-824b-82178c79b8fe None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-c2afe26f-dd71-47a3-824b-82178c79b8fe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:46.700179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:46.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:46.703540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:46.703540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:46.703540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:46.703540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:46.703540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 505/1032] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:46.707941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4516998a-a96f-452e-bdef-68083e3505cd None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:46.738288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:46.793477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07b408c-8c25-4b53-8491-5e189338061a None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:46.793749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07b408c-8c25-4b53-8491-5e189338061a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:46.794104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07b408c-8c25-4b53-8491-5e189338061a None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:46.796311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 506/1033] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:46.802244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edab8cd2-64d3-4b25-b4e8-0b5f4e242e9d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:46.802609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edab8cd2-64d3-4b25-b4e8-0b5f4e242e9d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:46.802893 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edab8cd2-64d3-4b25-b4e8-0b5f4e242e9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:46.824401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-edab8cd2-64d3-4b25-b4e8-0b5f4e242e9d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:46.825711 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-edab8cd2-64d3-4b25-b4e8-0b5f4e242e9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:46.842951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4516998a-a96f-452e-bdef-68083e3505cd None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:46.887837 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954fcc81-28f3-4800-9ed5-68b063ff29e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:46.888460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954fcc81-28f3-4800-9ed5-68b063ff29e3 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:46.889284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954fcc81-28f3-4800-9ed5-68b063ff29e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:46.889548 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954fcc81-28f3-4800-9ed5-68b063ff29e3 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:46.890254 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954fcc81-28f3-4800-9ed5-68b063ff29e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:46.890553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954fcc81-28f3-4800-9ed5-68b063ff29e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:46.891961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954fcc81-28f3-4800-9ed5-68b063ff29e3 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:46.909454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 528/1034] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:46.910569 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954fcc81-28f3-4800-9ed5-68b063ff29e3 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:46.916172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 507/1035] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:46 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:46.939843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edab8cd2-64d3-4b25-b4e8-0b5f4e242e9d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:47.012374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:47.031936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:47.037702 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954fcc81-28f3-4800-9ed5-68b063ff29e3 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:47.055884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edab8cd2-64d3-4b25-b4e8-0b5f4e242e9d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:47.079129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634b1377-abd8-43f1-859d-f1f0dc3b13d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:47.079758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634b1377-abd8-43f1-859d-f1f0dc3b13d5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:47.080036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634b1377-abd8-43f1-859d-f1f0dc3b13d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:47.080307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634b1377-abd8-43f1-859d-f1f0dc3b13d5 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:47.088305 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634b1377-abd8-43f1-859d-f1f0dc3b13d5 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:47.089757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634b1377-abd8-43f1-859d-f1f0dc3b13d5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:47.097253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2edf0c0-b926-48c0-a0e6-c0ef7dc35bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:47.097788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2edf0c0-b926-48c0-a0e6-c0ef7dc35bb8 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:47.098201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2edf0c0-b926-48c0-a0e6-c0ef7dc35bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:47.098361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2edf0c0-b926-48c0-a0e6-c0ef7dc35bb8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:47.099653 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2edf0c0-b926-48c0-a0e6-c0ef7dc35bb8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:47.101096 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2edf0c0-b926-48c0-a0e6-c0ef7dc35bb8 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:47.102958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 508/1036] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-634b1377-abd8-43f1-859d-f1f0dc3b13d5 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:47.105798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:47.108733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:47.108733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:47.108733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:47.108733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:47.108733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:47.108733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:47.108733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:47.108733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:47.108733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:55:47.108733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:47.109705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2edf0c0-b926-48c0-a0e6-c0ef7dc35bb8 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:47.111681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 529/1037] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Jul 31 00:55:46 2022] GET /identity/v3/users/nova => generated 82 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:47.114061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634b1377-abd8-43f1-859d-f1f0dc3b13d5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:47.233158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2edf0c0-b926-48c0-a0e6-c0ef7dc35bb8 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:47.233352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634b1377-abd8-43f1-859d-f1f0dc3b13d5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:47.271632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f60b7f-f960-4995-9301-81f8652da52c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:47.272304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f60b7f-f960-4995-9301-81f8652da52c None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:47.272563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f60b7f-f960-4995-9301-81f8652da52c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:47.272809 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f60b7f-f960-4995-9301-81f8652da52c None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:47.273460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f60b7f-f960-4995-9301-81f8652da52c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:47.274578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2a3d19-488b-4a20-8332-defe5a3f2f46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:47.275122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f60b7f-f960-4995-9301-81f8652da52c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:47.275257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2a3d19-488b-4a20-8332-defe5a3f2f46 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:47.275560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2a3d19-488b-4a20-8332-defe5a3f2f46 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:47.275815 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2a3d19-488b-4a20-8332-defe5a3f2f46 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:47.276558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2a3d19-488b-4a20-8332-defe5a3f2f46 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:47.276862 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2a3d19-488b-4a20-8332-defe5a3f2f46 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:47.278324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2a3d19-488b-4a20-8332-defe5a3f2f46 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:47.294842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:47.304375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 530/1038] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Jul 31 00:55:47 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:47.305666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2a3d19-488b-4a20-8332-defe5a3f2f46 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:47.309041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 509/1039] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:55:47 2022] POST /identity/v3/users => generated 285 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:47.427064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2a3d19-488b-4a20-8332-defe5a3f2f46 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:47.468329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b558aa8a-3dce-4178-a44c-ecc617fa040b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:47.468992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b558aa8a-3dce-4178-a44c-ecc617fa040b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:47.469272 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b558aa8a-3dce-4178-a44c-ecc617fa040b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:47.469546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b558aa8a-3dce-4178-a44c-ecc617fa040b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:47.553701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:47.558435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b558aa8a-3dce-4178-a44c-ecc617fa040b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:47.560034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b558aa8a-3dce-4178-a44c-ecc617fa040b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:47.691118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 531/1040] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:47 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:55:47.696815 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f60b7f-f960-4995-9301-81f8652da52c None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:47.702575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:47.837731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f60b7f-f960-4995-9301-81f8652da52c None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:47.894224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e696351-d75f-448b-9b9a-b2f66215c22f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:47.895297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e696351-d75f-448b-9b9a-b2f66215c22f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:47.895813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e696351-d75f-448b-9b9a-b2f66215c22f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:47.896311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e696351-d75f-448b-9b9a-b2f66215c22f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:47.915972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e696351-d75f-448b-9b9a-b2f66215c22f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:47.918897 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e696351-d75f-448b-9b9a-b2f66215c22f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-9e696351-d75f-448b-9b9a-b2f66215c22f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:47.947777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:47.948751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:47.949646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:47.949646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:47.949646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:47.949646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:47.955121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 510/1041] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:47.956302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b558aa8a-3dce-4178-a44c-ecc617fa040b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:48.061936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:48.066099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b558aa8a-3dce-4178-a44c-ecc617fa040b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:48.104513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bb417c-fcd2-452a-ab31-90f0eedeb25c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:48.105066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bb417c-fcd2-452a-ab31-90f0eedeb25c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:48.105340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bb417c-fcd2-452a-ab31-90f0eedeb25c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:48.105611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bb417c-fcd2-452a-ab31-90f0eedeb25c None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:48.106312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bb417c-fcd2-452a-ab31-90f0eedeb25c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:48.106614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bb417c-fcd2-452a-ab31-90f0eedeb25c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:48.108055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bb417c-fcd2-452a-ab31-90f0eedeb25c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:48.125409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 532/1042] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:48.127272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e696351-d75f-448b-9b9a-b2f66215c22f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:48.190860 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:48.239869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e696351-d75f-448b-9b9a-b2f66215c22f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:48.276285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5fd4b4-63bc-4d1d-af7a-d09053087b29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:48.277012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5fd4b4-63bc-4d1d-af7a-d09053087b29 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:48.277268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5fd4b4-63bc-4d1d-af7a-d09053087b29 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:48.277472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5fd4b4-63bc-4d1d-af7a-d09053087b29 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:48.285271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5fd4b4-63bc-4d1d-af7a-d09053087b29 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:48.286730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5fd4b4-63bc-4d1d-af7a-d09053087b29 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-8d5fd4b4-63bc-4d1d-af7a-d09053087b29 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:48.298610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:48.299539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:48.306072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:48.306072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:48.306072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:48.306072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:48.306072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:48.306072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:48.306072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:48.306072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:48.306072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 511/1043] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:48.307705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bb417c-fcd2-452a-ab31-90f0eedeb25c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:48.418330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bb417c-fcd2-452a-ab31-90f0eedeb25c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:48.421504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1debe29d-45a6-4ef0-854a-91cbcc258118 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:48.421972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1debe29d-45a6-4ef0-854a-91cbcc258118 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:48.422454 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1debe29d-45a6-4ef0-854a-91cbcc258118 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:48.425105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 512/1044] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:48.431696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2ffbf1-9491-43be-8164-bc8e866e3bdf None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:48.431994 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2ffbf1-9491-43be-8164-bc8e866e3bdf None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:48.432253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2ffbf1-9491-43be-8164-bc8e866e3bdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:48.453114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-eb2ffbf1-9491-43be-8164-bc8e866e3bdf None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:48.454973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-eb2ffbf1-9491-43be-8164-bc8e866e3bdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:48.459306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8a93e4-1a09-46b2-941c-79c174273c8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:48.459978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8a93e4-1a09-46b2-941c-79c174273c8c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:48.460260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8a93e4-1a09-46b2-941c-79c174273c8c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:48.460536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8a93e4-1a09-46b2-941c-79c174273c8c None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:48.461292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8a93e4-1a09-46b2-941c-79c174273c8c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:48.461743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8a93e4-1a09-46b2-941c-79c174273c8c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:48.463199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8a93e4-1a09-46b2-941c-79c174273c8c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:48.477050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 533/1045] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:48.483337 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8a93e4-1a09-46b2-941c-79c174273c8c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:48.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:48.551392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 513/1046] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:48 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:48.572440 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2ffbf1-9491-43be-8164-bc8e866e3bdf None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:48.606781 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8a93e4-1a09-46b2-941c-79c174273c8c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:48.625421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:48.647476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c544cd-3217-4deb-a891-5d92fa8332dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:48.648089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c544cd-3217-4deb-a891-5d92fa8332dd None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:48.648366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c544cd-3217-4deb-a891-5d92fa8332dd None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:48.648635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c544cd-3217-4deb-a891-5d92fa8332dd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:48.649425 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c544cd-3217-4deb-a891-5d92fa8332dd None admin] RBAC: Adding query filter params (role.id=53d929a69ffa476eac93c56d7de72a16, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=5b0bd572ae7a4c7f9e07fe0b9708aa7e) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:48.649730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c544cd-3217-4deb-a891-5d92fa8332dd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:48.651161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c544cd-3217-4deb-a891-5d92fa8332dd None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:48.657817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 534/1047] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:48 2022] GET /identity/v3/role_assignments?user.id=5b0bd572ae7a4c7f9e07fe0b9708aa7e&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=53d929a69ffa476eac93c56d7de72a16 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:48.690252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2ffbf1-9491-43be-8164-bc8e866e3bdf None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:48.731057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24606e3c-045f-471c-a915-8214fce84c88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:48.731744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24606e3c-045f-471c-a915-8214fce84c88 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:48.732019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24606e3c-045f-471c-a915-8214fce84c88 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:48.732327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24606e3c-045f-471c-a915-8214fce84c88 None admin] PATH_INFO: `/v3/services/image` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:48.732955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24606e3c-045f-471c-a915-8214fce84c88 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:48.734360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24606e3c-045f-471c-a915-8214fce84c88 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-24606e3c-045f-471c-a915-8214fce84c88 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:48.746703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:48.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:48.749420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 514/1048] 10.222.0.42 () {56 vars in 1105 bytes} [Sun Jul 31 00:55:48 2022] GET /identity/v3/services/image => generated 86 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:48.755385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c544cd-3217-4deb-a891-5d92fa8332dd None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:48.865180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:48.883857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c544cd-3217-4deb-a891-5d92fa8332dd None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:48.925791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf22b842-bc63-41ba-965e-999972ba17ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:48.926414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf22b842-bc63-41ba-965e-999972ba17ff None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:48.926695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf22b842-bc63-41ba-965e-999972ba17ff None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:48.926966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf22b842-bc63-41ba-965e-999972ba17ff None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:48.927785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf22b842-bc63-41ba-965e-999972ba17ff None admin] RBAC: Adding query filter params (name=image) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:48.928099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf22b842-bc63-41ba-965e-999972ba17ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:48.929493 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf22b842-bc63-41ba-965e-999972ba17ff None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:48.936051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 535/1049] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:48 2022] GET /identity/v3/services?name=image => generated 122 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:48.942019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24606e3c-045f-471c-a915-8214fce84c88 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:49.053432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24606e3c-045f-471c-a915-8214fce84c88 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:49.065404 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:49.090444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02fa2c8-2ecd-47bf-9f62-2fbbceb7191a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:49.091064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02fa2c8-2ecd-47bf-9f62-2fbbceb7191a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:49.091341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02fa2c8-2ecd-47bf-9f62-2fbbceb7191a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:49.091704 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02fa2c8-2ecd-47bf-9f62-2fbbceb7191a None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:49.092384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02fa2c8-2ecd-47bf-9f62-2fbbceb7191a None admin] RBAC: Adding query filter params (type=image) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:49.092684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02fa2c8-2ecd-47bf-9f62-2fbbceb7191a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:49.094053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02fa2c8-2ecd-47bf-9f62-2fbbceb7191a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:49.100779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 515/1050] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:48 2022] GET /identity/v3/services?type=image => generated 350 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:49.108247 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf22b842-bc63-41ba-965e-999972ba17ff None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:49.225071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf22b842-bc63-41ba-965e-999972ba17ff None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:49.264372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58f5528-2079-450f-aced-e539a47cb7eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:49.265040 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58f5528-2079-450f-aced-e539a47cb7eb None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:49.265315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58f5528-2079-450f-aced-e539a47cb7eb None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:49.265583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58f5528-2079-450f-aced-e539a47cb7eb None admin] PATH_INFO: `/v3/endpoints` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:49.266330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58f5528-2079-450f-aced-e539a47cb7eb None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:49.267767 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58f5528-2079-450f-aced-e539a47cb7eb None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:49.292298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 536/1051] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 00:55:49 2022] POST /identity/v3/endpoints => generated 327 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:55:49.520380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e6c907-24c4-4f0a-bf8d-c978292c0892 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:49.520937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e6c907-24c4-4f0a-bf8d-c978292c0892 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:49.520937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e6c907-24c4-4f0a-bf8d-c978292c0892 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:49.524176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 516/1052] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:49.532091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f0d500-83b9-4649-a2d7-35df8d4776e2 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:49.532615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f0d500-83b9-4649-a2d7-35df8d4776e2 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:49.533046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f0d500-83b9-4649-a2d7-35df8d4776e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:49.542018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:49.555616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-19f0d500-83b9-4649-a2d7-35df8d4776e2 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:49.557415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-19f0d500-83b9-4649-a2d7-35df8d4776e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:49.647087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 537/1053] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:49 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:49.672300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e6c907-24c4-4f0a-bf8d-c978292c0892 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:49.781615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e6c907-24c4-4f0a-bf8d-c978292c0892 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:49.819210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebed15c4-5ba6-4045-b25b-75955a4e6423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:49.819887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebed15c4-5ba6-4045-b25b-75955a4e6423 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:49.820172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebed15c4-5ba6-4045-b25b-75955a4e6423 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:49.820444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebed15c4-5ba6-4045-b25b-75955a4e6423 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:49.832077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebed15c4-5ba6-4045-b25b-75955a4e6423 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-ebed15c4-5ba6-4045-b25b-75955a4e6423 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:49.837098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:49.838106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:49.839013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:49.839013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:49.839013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:49.839013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:49.839013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:49.839013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:49.839013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:49.839013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:49.839013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:49.839013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:49.839013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:49.839013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:49.843449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 517/1054] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:49.845361 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f0d500-83b9-4649-a2d7-35df8d4776e2 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:49.952963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f0d500-83b9-4649-a2d7-35df8d4776e2 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:49.984832 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:49.992547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b171e0-bb45-4a59-bc9f-e58b9f3f7adc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:49.993176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b171e0-bb45-4a59-bc9f-e58b9f3f7adc None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:49.993450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b171e0-bb45-4a59-bc9f-e58b9f3f7adc None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:49.993720 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b171e0-bb45-4a59-bc9f-e58b9f3f7adc None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:49.994490 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b171e0-bb45-4a59-bc9f-e58b9f3f7adc None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:49.994809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b171e0-bb45-4a59-bc9f-e58b9f3f7adc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:49.996141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b171e0-bb45-4a59-bc9f-e58b9f3f7adc None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:50.005340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 538/1055] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:55:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:50.010919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebed15c4-5ba6-4045-b25b-75955a4e6423 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:50.051094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:50.122012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebed15c4-5ba6-4045-b25b-75955a4e6423 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:50.159179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23f70d9-2cab-4f56-bbfd-9683d20b0025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:50.159847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23f70d9-2cab-4f56-bbfd-9683d20b0025 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:50.160130 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23f70d9-2cab-4f56-bbfd-9683d20b0025 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:50.160402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23f70d9-2cab-4f56-bbfd-9683d20b0025 None admin] PATH_INFO: `/v3/users/1ab7192596894bd2bebd3a28316cec06` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:50.174223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23f70d9-2cab-4f56-bbfd-9683d20b0025 None admin] RBAC: Authorizing `identity:get_user(user_id=1ab7192596894bd2bebd3a28316cec06)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:50.175969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23f70d9-2cab-4f56-bbfd-9683d20b0025 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:50.182357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 518/1056] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:55:50 2022] GET /identity/v3/users/1ab7192596894bd2bebd3a28316cec06 => generated 285 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:50.183658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b171e0-bb45-4a59-bc9f-e58b9f3f7adc None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:50.294577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b171e0-bb45-4a59-bc9f-e58b9f3f7adc None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:50.330421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39de9b2-e9f3-4377-b964-b7c07b56507b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:50.330960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39de9b2-e9f3-4377-b964-b7c07b56507b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:50.331238 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39de9b2-e9f3-4377-b964-b7c07b56507b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:50.331543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39de9b2-e9f3-4377-b964-b7c07b56507b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:50.391234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:50.402629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39de9b2-e9f3-4377-b964-b7c07b56507b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:50.403991 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39de9b2-e9f3-4377-b964-b7c07b56507b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:50.517681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 539/1057] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:50 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:55:50.523609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23f70d9-2cab-4f56-bbfd-9683d20b0025 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:50.634757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:50.646057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23f70d9-2cab-4f56-bbfd-9683d20b0025 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:50.691061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868bfd1a-f480-48c1-8eef-b7bbdcda7da2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:50.691756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868bfd1a-f480-48c1-8eef-b7bbdcda7da2 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:50.692032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868bfd1a-f480-48c1-8eef-b7bbdcda7da2 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:50.692302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868bfd1a-f480-48c1-8eef-b7bbdcda7da2 None admin] PATH_INFO: `/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:50.693457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868bfd1a-f480-48c1-8eef-b7bbdcda7da2 None admin] RBAC: Authorizing `identity:get_project(project_id=df1ae0d2fe6d4e89b266674bfd40f1f4)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:50.694880 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868bfd1a-f480-48c1-8eef-b7bbdcda7da2 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:50.701542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 519/1058] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:55:50 2022] GET /identity/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4 => generated 311 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:50.702954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39de9b2-e9f3-4377-b964-b7c07b56507b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:50.798174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:50.808798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39de9b2-e9f3-4377-b964-b7c07b56507b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:50.845176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c5fe25-e617-4a96-add6-aaede486ec7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:50.845793 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c5fe25-e617-4a96-add6-aaede486ec7f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:50.846070 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c5fe25-e617-4a96-add6-aaede486ec7f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:50.846342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c5fe25-e617-4a96-add6-aaede486ec7f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:50.847128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c5fe25-e617-4a96-add6-aaede486ec7f None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=df1ae0d2fe6d4e89b266674bfd40f1f4, user.id=1ab7192596894bd2bebd3a28316cec06) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:50.847461 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c5fe25-e617-4a96-add6-aaede486ec7f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:50.848904 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c5fe25-e617-4a96-add6-aaede486ec7f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:50.863396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 540/1059] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:50 2022] GET /identity/v3/role_assignments?user.id=1ab7192596894bd2bebd3a28316cec06&scope.project.id=df1ae0d2fe6d4e89b266674bfd40f1f4&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:50.903620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0f1206-fa5e-4b27-90e4-20e13a79aac4 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:50.903620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0f1206-fa5e-4b27-90e4-20e13a79aac4 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:50.903620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0f1206-fa5e-4b27-90e4-20e13a79aac4 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:50.906629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 520/1060] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:50 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:50.911648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399dfc99-ab0e-4639-9a59-774ddcbe880d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:50.911966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399dfc99-ab0e-4639-9a59-774ddcbe880d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:50.912203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399dfc99-ab0e-4639-9a59-774ddcbe880d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:50.933198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-399dfc99-ab0e-4639-9a59-774ddcbe880d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:50.934437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-399dfc99-ab0e-4639-9a59-774ddcbe880d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:51.010491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 541/1061] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:50 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:51.034106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0f1206-fa5e-4b27-90e4-20e13a79aac4 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:51.148927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0f1206-fa5e-4b27-90e4-20e13a79aac4 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:51.185537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba51a77-cf39-4d0c-891a-97f0486d76bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:51.186075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba51a77-cf39-4d0c-891a-97f0486d76bf None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:51.186355 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba51a77-cf39-4d0c-891a-97f0486d76bf None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:51.186634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba51a77-cf39-4d0c-891a-97f0486d76bf None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:51.197236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba51a77-cf39-4d0c-891a-97f0486d76bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-8ba51a77-cf39-4d0c-891a-97f0486d76bf None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:51.202757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:51.203772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:51.210367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 521/1062] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Jul 31 00:55:51 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:51.211077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399dfc99-ab0e-4639-9a59-774ddcbe880d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:51.304162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:51.321159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399dfc99-ab0e-4639-9a59-774ddcbe880d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:51.374191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46f1e76-aa4e-4a61-aac0-d24e13e462c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:51.375699 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46f1e76-aa4e-4a61-aac0-d24e13e462c0 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:51.376443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46f1e76-aa4e-4a61-aac0-d24e13e462c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:51.377156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46f1e76-aa4e-4a61-aac0-d24e13e462c0 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:51.378595 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46f1e76-aa4e-4a61-aac0-d24e13e462c0 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:51.379341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46f1e76-aa4e-4a61-aac0-d24e13e462c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:51.381949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46f1e76-aa4e-4a61-aac0-d24e13e462c0 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:51.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17cfed0-3e3a-407d-8a72-1d26cf6ad4ba None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:51.392584 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17cfed0-3e3a-407d-8a72-1d26cf6ad4ba None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:51.392868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17cfed0-3e3a-407d-8a72-1d26cf6ad4ba None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:51.394830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 522/1063] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:51.396455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 542/1064] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Jul 31 00:55:51 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:51.400765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9c8a78-db92-41cc-b815-c97864cca303 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:51.401113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9c8a78-db92-41cc-b815-c97864cca303 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:51.401476 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9c8a78-db92-41cc-b815-c97864cca303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:51.402636 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46f1e76-aa4e-4a61-aac0-d24e13e462c0 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:51.428451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-2c9c8a78-db92-41cc-b815-c97864cca303 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:51.429842 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2c9c8a78-db92-41cc-b815-c97864cca303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:51.448108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:51.549549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 523/1065] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:51 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:51.550448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46f1e76-aa4e-4a61-aac0-d24e13e462c0 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:51.568323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9c8a78-db92-41cc-b815-c97864cca303 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:51.593531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d184d992-8349-4feb-8257-620780a2feb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:51.594220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d184d992-8349-4feb-8257-620780a2feb1 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:51.594528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d184d992-8349-4feb-8257-620780a2feb1 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:51.595631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d184d992-8349-4feb-8257-620780a2feb1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:51.608843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d184d992-8349-4feb-8257-620780a2feb1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:51.610328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d184d992-8349-4feb-8257-620780a2feb1 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:51.619541 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-d184d992-8349-4feb-8257-620780a2feb1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:51.627415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:51.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:51.629715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:51.629715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:51.629715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:51.629715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:51.629715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 543/1066] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:51.634009 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d184d992-8349-4feb-8257-620780a2feb1 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:51.724931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9c8a78-db92-41cc-b815-c97864cca303 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:51.764797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa7edc6-6b9c-4997-abec-5ce93ed568d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:51.765448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7edc6-6b9c-4997-abec-5ce93ed568d3 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:51.765686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d184d992-8349-4feb-8257-620780a2feb1 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:51.766207 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7edc6-6b9c-4997-abec-5ce93ed568d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:51.766207 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7edc6-6b9c-4997-abec-5ce93ed568d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:51.789402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:51.807051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558d3013-e899-4cd4-b602-3873db5f2ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:51.808062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558d3013-e899-4cd4-b602-3873db5f2ebe None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:51.808346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558d3013-e899-4cd4-b602-3873db5f2ebe None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:51.808909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558d3013-e899-4cd4-b602-3873db5f2ebe None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:51.809610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558d3013-e899-4cd4-b602-3873db5f2ebe None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:51.809926 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558d3013-e899-4cd4-b602-3873db5f2ebe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:51.811374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558d3013-e899-4cd4-b602-3873db5f2ebe None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:51.830111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 544/1067] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:51.830949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558d3013-e899-4cd4-b602-3873db5f2ebe None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:51.856574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa7edc6-6b9c-4997-abec-5ce93ed568d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:51.859020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa7edc6-6b9c-4997-abec-5ce93ed568d3 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:51.903309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:51.957055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558d3013-e899-4cd4-b602-3873db5f2ebe None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:51.979478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 524/1068] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:51 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 31 00:55:51.982537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa7edc6-6b9c-4997-abec-5ce93ed568d3 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:52.000275 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2c1eb9-73fa-4ded-85b9-0b3405a8397b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:52.000808 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2c1eb9-73fa-4ded-85b9-0b3405a8397b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:52.001090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2c1eb9-73fa-4ded-85b9-0b3405a8397b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:52.001646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2c1eb9-73fa-4ded-85b9-0b3405a8397b None admin] PATH_INFO: `/v3/users/nova` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:52.009461 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2c1eb9-73fa-4ded-85b9-0b3405a8397b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:52.010960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2c1eb9-73fa-4ded-85b9-0b3405a8397b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-4a2c1eb9-73fa-4ded-85b9-0b3405a8397b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:52.025553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:52.026758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:52.027768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:52.027768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:52.027768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:52.027768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:52.027768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:52.027768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:52.027768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:52.027768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:52.027768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:55:52.027768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:52.027768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 545/1069] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Jul 31 00:55:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:52.033549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2c1eb9-73fa-4ded-85b9-0b3405a8397b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:52.054167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:52.122204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa7edc6-6b9c-4997-abec-5ce93ed568d3 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:52.160213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2c1eb9-73fa-4ded-85b9-0b3405a8397b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:52.160623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d62be8-6d0e-490b-bdf4-0e65a45c8ab3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:52.161292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d62be8-6d0e-490b-bdf4-0e65a45c8ab3 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:52.161574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d62be8-6d0e-490b-bdf4-0e65a45c8ab3 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:52.161884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d62be8-6d0e-490b-bdf4-0e65a45c8ab3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:52.173524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d62be8-6d0e-490b-bdf4-0e65a45c8ab3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:52.175046 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d62be8-6d0e-490b-bdf4-0e65a45c8ab3 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:52.186098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-d6d62be8-6d0e-490b-bdf4-0e65a45c8ab3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:52.193558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:52.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:52.195594 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:52.195594 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:52.195594 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:52.195594 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:52.195594 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 525/1070] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:52.201907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d62be8-6d0e-490b-bdf4-0e65a45c8ab3 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:52.214181 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143dc389-318c-4185-ba2e-3f923339066c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:52.214814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143dc389-318c-4185-ba2e-3f923339066c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:52.215091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143dc389-318c-4185-ba2e-3f923339066c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:52.215389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143dc389-318c-4185-ba2e-3f923339066c None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:52.216199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143dc389-318c-4185-ba2e-3f923339066c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:52.217082 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143dc389-318c-4185-ba2e-3f923339066c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:52.218601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143dc389-318c-4185-ba2e-3f923339066c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:52.244040 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 546/1071] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Jul 31 00:55:52 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:52.249795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143dc389-318c-4185-ba2e-3f923339066c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:52.333961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d62be8-6d0e-490b-bdf4-0e65a45c8ab3 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:52.336942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:52.384078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1e24b1-c1bb-47ff-9e8e-c11e4bb635b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:52.384704 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1e24b1-c1bb-47ff-9e8e-c11e4bb635b9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:52.384883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143dc389-318c-4185-ba2e-3f923339066c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:52.384997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1e24b1-c1bb-47ff-9e8e-c11e4bb635b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:52.385247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1e24b1-c1bb-47ff-9e8e-c11e4bb635b9 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:52.385942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1e24b1-c1bb-47ff-9e8e-c11e4bb635b9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:52.386246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1e24b1-c1bb-47ff-9e8e-c11e4bb635b9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:52.387649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1e24b1-c1bb-47ff-9e8e-c11e4bb635b9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:52.411386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 526/1072] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:52.451912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab90ead-a4c0-4802-b240-885dfdabd764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:52.453026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab90ead-a4c0-4802-b240-885dfdabd764 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:52.453499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab90ead-a4c0-4802-b240-885dfdabd764 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:52.453966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab90ead-a4c0-4802-b240-885dfdabd764 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:52.473832 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab90ead-a4c0-4802-b240-885dfdabd764 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:52.476566 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab90ead-a4c0-4802-b240-885dfdabd764 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-3ab90ead-a4c0-4802-b240-885dfdabd764 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:52.498551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:52.499568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:52.501250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:52.501250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:52.501250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:52.501250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:52.503898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 547/1073] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:52.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1e24b1-c1bb-47ff-9e8e-c11e4bb635b9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:52.640309 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1e24b1-c1bb-47ff-9e8e-c11e4bb635b9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:52.683304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0bc0fc-eda4-4079-93d3-106c32da42f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:52.683973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0bc0fc-eda4-4079-93d3-106c32da42f9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:52.684248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0bc0fc-eda4-4079-93d3-106c32da42f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:52.684517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0bc0fc-eda4-4079-93d3-106c32da42f9 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:52.685425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0bc0fc-eda4-4079-93d3-106c32da42f9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:52.685857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0bc0fc-eda4-4079-93d3-106c32da42f9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:52.687478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0bc0fc-eda4-4079-93d3-106c32da42f9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:52.700726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 527/1074] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:52.707161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab90ead-a4c0-4802-b240-885dfdabd764 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:52.746882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:52.817898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab90ead-a4c0-4802-b240-885dfdabd764 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:52.852570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c3808c-deb9-4233-9757-6666251c528f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:52.853209 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c3808c-deb9-4233-9757-6666251c528f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:52.853485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c3808c-deb9-4233-9757-6666251c528f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:52.853754 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c3808c-deb9-4233-9757-6666251c528f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:52.861104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c3808c-deb9-4233-9757-6666251c528f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:52.862560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c3808c-deb9-4233-9757-6666251c528f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-64c3808c-deb9-4233-9757-6666251c528f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:52.874259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:52.875229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:52.876154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:52.876154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:52.876154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:52.876154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:52.876154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:52.876154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:52.876154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:52.876154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:52.876154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 548/1075] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:52.881388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0bc0fc-eda4-4079-93d3-106c32da42f9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:52.907873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:52.951336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b04f01-4d41-4dac-add5-884aa39c71fd None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:52.951653 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b04f01-4d41-4dac-add5-884aa39c71fd None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:52.951936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b04f01-4d41-4dac-add5-884aa39c71fd None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:52.953400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 549/1076] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:55:52.958605 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd651f7e-1b06-4839-9b13-d74b438d5a2d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:52.958900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd651f7e-1b06-4839-9b13-d74b438d5a2d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:52.959139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd651f7e-1b06-4839-9b13-d74b438d5a2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:52.978094 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-fd651f7e-1b06-4839-9b13-d74b438d5a2d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:52.979301 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-fd651f7e-1b06-4839-9b13-d74b438d5a2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:52.986412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0bc0fc-eda4-4079-93d3-106c32da42f9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:53.021828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1756af2e-4436-458f-b245-100ab3e93024 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:53.022443 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1756af2e-4436-458f-b245-100ab3e93024 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:53.022722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1756af2e-4436-458f-b245-100ab3e93024 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:53.022993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1756af2e-4436-458f-b245-100ab3e93024 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:53.023768 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1756af2e-4436-458f-b245-100ab3e93024 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:53.024073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1756af2e-4436-458f-b245-100ab3e93024 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:53.025506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1756af2e-4436-458f-b245-100ab3e93024 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:53.038305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 528/1077] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:53.044256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1756af2e-4436-458f-b245-100ab3e93024 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:53.049034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 550/1078] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:52 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:53.071811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd651f7e-1b06-4839-9b13-d74b438d5a2d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:53.093210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:53.131514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:53.151041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1756af2e-4436-458f-b245-100ab3e93024 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:53.174739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd651f7e-1b06-4839-9b13-d74b438d5a2d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:53.185804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5ba254-8002-4150-8e84-dcff62e0f74e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:53.186467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5ba254-8002-4150-8e84-dcff62e0f74e None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:53.186743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5ba254-8002-4150-8e84-dcff62e0f74e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:53.187012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5ba254-8002-4150-8e84-dcff62e0f74e None admin] PATH_INFO: `/v3/projects/8b41573c9f3146d19a306089d686015c/users/5b0bd572ae7a4c7f9e07fe0b9708aa7e/roles/53d929a69ffa476eac93c56d7de72a16` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:53.189242 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5ba254-8002-4150-8e84-dcff62e0f74e None admin] RBAC: Authorizing `identity:create_grant(project_id=8b41573c9f3146d19a306089d686015c, user_id=5b0bd572ae7a4c7f9e07fe0b9708aa7e, role_id=53d929a69ffa476eac93c56d7de72a16)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:53.190670 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5ba254-8002-4150-8e84-dcff62e0f74e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:53.204320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 529/1079] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:55:53 2022] PUT /identity/v3/projects/8b41573c9f3146d19a306089d686015c/users/5b0bd572ae7a4c7f9e07fe0b9708aa7e/roles/53d929a69ffa476eac93c56d7de72a16 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 00:55:53.215207 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9e709d-5c1a-43d0-975e-35b95c10e753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:53.215962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9e709d-5c1a-43d0-975e-35b95c10e753 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:53.216241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9e709d-5c1a-43d0-975e-35b95c10e753 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:53.216558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9e709d-5c1a-43d0-975e-35b95c10e753 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:53.228974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9e709d-5c1a-43d0-975e-35b95c10e753 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-7d9e709d-5c1a-43d0-975e-35b95c10e753 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:53.234098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:53.235391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:53.236602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:53.239560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 551/1080] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:53.241581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5ba254-8002-4150-8e84-dcff62e0f74e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:53.359075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5ba254-8002-4150-8e84-dcff62e0f74e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:53.394947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc63c38-bc6a-4449-aa53-89b79aed81e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:53.395596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc63c38-bc6a-4449-aa53-89b79aed81e6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:53.395893 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc63c38-bc6a-4449-aa53-89b79aed81e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:53.396137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc63c38-bc6a-4449-aa53-89b79aed81e6 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:53.396840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc63c38-bc6a-4449-aa53-89b79aed81e6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:53.398301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc63c38-bc6a-4449-aa53-89b79aed81e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:53.398493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc63c38-bc6a-4449-aa53-89b79aed81e6 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:53.407652 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 530/1081] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:55:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:53.413652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9e709d-5c1a-43d0-975e-35b95c10e753 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:53.496693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:53.517116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9e709d-5c1a-43d0-975e-35b95c10e753 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:53.556282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138392fd-c87a-4de2-bbf1-965a0fbab33b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:53.556959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138392fd-c87a-4de2-bbf1-965a0fbab33b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:53.557239 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138392fd-c87a-4de2-bbf1-965a0fbab33b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:53.557518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138392fd-c87a-4de2-bbf1-965a0fbab33b None admin] PATH_INFO: `/v3/users/1ab7192596894bd2bebd3a28316cec06` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:53.558683 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138392fd-c87a-4de2-bbf1-965a0fbab33b None admin] RBAC: Authorizing `identity:get_user(user_id=1ab7192596894bd2bebd3a28316cec06)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:53.560765 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138392fd-c87a-4de2-bbf1-965a0fbab33b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:53.562403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 552/1082] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:55:53 2022] GET /identity/v3/users/1ab7192596894bd2bebd3a28316cec06 => generated 285 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:53.568153 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc63c38-bc6a-4449-aa53-89b79aed81e6 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:53.572689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:53.684862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc63c38-bc6a-4449-aa53-89b79aed81e6 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:53.722907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617879ba-5082-4d4d-bd9c-e65c2f7b9749 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:53.723587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617879ba-5082-4d4d-bd9c-e65c2f7b9749 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:53.723870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617879ba-5082-4d4d-bd9c-e65c2f7b9749 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:53.724144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617879ba-5082-4d4d-bd9c-e65c2f7b9749 None admin] PATH_INFO: `/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:53.725353 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617879ba-5082-4d4d-bd9c-e65c2f7b9749 None admin] RBAC: Authorizing `identity:get_project(project_id=df1ae0d2fe6d4e89b266674bfd40f1f4)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:53.726819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617879ba-5082-4d4d-bd9c-e65c2f7b9749 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:53.728831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 531/1083] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:55:53 2022] GET /identity/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4 => generated 311 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:53.735504 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138392fd-c87a-4de2-bbf1-965a0fbab33b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:53.843993 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138392fd-c87a-4de2-bbf1-965a0fbab33b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:53.878661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76df671-81cb-4f3f-9ef9-30326e8cc496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:53.879327 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76df671-81cb-4f3f-9ef9-30326e8cc496 None admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:53.879623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76df671-81cb-4f3f-9ef9-30326e8cc496 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:53.879908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76df671-81cb-4f3f-9ef9-30326e8cc496 None admin] PATH_INFO: `/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4/users/1ab7192596894bd2bebd3a28316cec06/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:53.881830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76df671-81cb-4f3f-9ef9-30326e8cc496 None admin] RBAC: Authorizing `identity:create_grant(project_id=df1ae0d2fe6d4e89b266674bfd40f1f4, user_id=1ab7192596894bd2bebd3a28316cec06, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:53.883246 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76df671-81cb-4f3f-9ef9-30326e8cc496 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:53.896763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 553/1084] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:55:53 2022] PUT /identity/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4/users/1ab7192596894bd2bebd3a28316cec06/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:55:55.225052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90862859-6016-4086-9c74-315ac7a7582c None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:55.225615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90862859-6016-4086-9c74-315ac7a7582c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:55.225615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90862859-6016-4086-9c74-315ac7a7582c None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:55.228121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 532/1085] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:55.235342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3768af-400c-49e5-9e52-81e5e6f443e0 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:55.235954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3768af-400c-49e5-9e52-81e5e6f443e0 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:55.235954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3768af-400c-49e5-9e52-81e5e6f443e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:55.254398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-fc3768af-400c-49e5-9e52-81e5e6f443e0 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:55.255625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-fc3768af-400c-49e5-9e52-81e5e6f443e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:55.257920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:55.328909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 554/1086] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:55 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:55.351478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90862859-6016-4086-9c74-315ac7a7582c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:55.424864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:55.470661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90862859-6016-4086-9c74-315ac7a7582c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:55.508444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c2d44a-5861-400b-8891-d9e2845ad068 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:55.509003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c2d44a-5861-400b-8891-d9e2845ad068 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:55.509281 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c2d44a-5861-400b-8891-d9e2845ad068 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:55.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c2d44a-5861-400b-8891-d9e2845ad068 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:55.520901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c2d44a-5861-400b-8891-d9e2845ad068 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-c5c2d44a-5861-400b-8891-d9e2845ad068 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:55.525913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:55.526951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:55.527977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 533/1087] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Jul 31 00:55:55 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:55.533232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3768af-400c-49e5-9e52-81e5e6f443e0 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:55.643144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3768af-400c-49e5-9e52-81e5e6f443e0 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:55.683024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a2dc5a-8911-4f6c-ae97-96ad2d15dc6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:55.683710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a2dc5a-8911-4f6c-ae97-96ad2d15dc6b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:55.683988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a2dc5a-8911-4f6c-ae97-96ad2d15dc6b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:55.684258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a2dc5a-8911-4f6c-ae97-96ad2d15dc6b None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:55.685545 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a2dc5a-8911-4f6c-ae97-96ad2d15dc6b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:55.685855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a2dc5a-8911-4f6c-ae97-96ad2d15dc6b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:55.687274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a2dc5a-8911-4f6c-ae97-96ad2d15dc6b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:55.693255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:55.697751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 555/1088] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Jul 31 00:55:55 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:55.704262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c2d44a-5861-400b-8891-d9e2845ad068 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:55.818448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c2d44a-5861-400b-8891-d9e2845ad068 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:55.853842 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:55.861088 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67d166e-0168-418f-be2f-140f72713001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:55.861744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67d166e-0168-418f-be2f-140f72713001 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:55.862016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67d166e-0168-418f-be2f-140f72713001 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:55.862283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67d166e-0168-418f-be2f-140f72713001 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:55.874579 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67d166e-0168-418f-be2f-140f72713001 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:55.876083 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67d166e-0168-418f-be2f-140f72713001 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-d67d166e-0168-418f-be2f-140f72713001 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:55.892183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:55.893289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:55.894178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:55.894178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:55.894178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:55.894178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:55.894178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 534/1089] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:55:55.901143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a2dc5a-8911-4f6c-ae97-96ad2d15dc6b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:55.934298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19f316a-3b5f-4333-a115-092daefbb603 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:55.934568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19f316a-3b5f-4333-a115-092daefbb603 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:55.934836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19f316a-3b5f-4333-a115-092daefbb603 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:55.937179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 535/1090] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:55:55.942523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6921ed29-4fbc-445a-af32-42ae82e1365c None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:55.942800 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6921ed29-4fbc-445a-af32-42ae82e1365c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:55.943074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6921ed29-4fbc-445a-af32-42ae82e1365c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:55.962470 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6921ed29-4fbc-445a-af32-42ae82e1365c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:55.963712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6921ed29-4fbc-445a-af32-42ae82e1365c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:56.016479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a2dc5a-8911-4f6c-ae97-96ad2d15dc6b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:56.048141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 536/1091] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:55 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:55:56.055972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f436af-56ce-4b5d-a4b6-609ec4121e6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:56.056594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f436af-56ce-4b5d-a4b6-609ec4121e6b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:56.061770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f436af-56ce-4b5d-a4b6-609ec4121e6b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:56.061770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f436af-56ce-4b5d-a4b6-609ec4121e6b None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:56.061770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f436af-56ce-4b5d-a4b6-609ec4121e6b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:56.061770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f436af-56ce-4b5d-a4b6-609ec4121e6b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:56.061770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f436af-56ce-4b5d-a4b6-609ec4121e6b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:56.072304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6921ed29-4fbc-445a-af32-42ae82e1365c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:56.081427 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 556/1092] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:56.087256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f436af-56ce-4b5d-a4b6-609ec4121e6b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:56.175752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:56.184155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6921ed29-4fbc-445a-af32-42ae82e1365c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:56.198587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:56.220830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d30a355-48db-4c16-b0ea-68ab6e26f627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:56.221475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d30a355-48db-4c16-b0ea-68ab6e26f627 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:56.221764 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d30a355-48db-4c16-b0ea-68ab6e26f627 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:56.222097 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d30a355-48db-4c16-b0ea-68ab6e26f627 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:56.228414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f436af-56ce-4b5d-a4b6-609ec4121e6b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:56.234225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d30a355-48db-4c16-b0ea-68ab6e26f627 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-4d30a355-48db-4c16-b0ea-68ab6e26f627 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:55:56.239416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:56.240585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:55:56.242184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 537/1093] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:55:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:56.247202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d30a355-48db-4c16-b0ea-68ab6e26f627 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:56.267657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb8f63b-c2df-414f-891b-16b5bda02012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:56.268294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb8f63b-c2df-414f-891b-16b5bda02012 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:56.268569 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb8f63b-c2df-414f-891b-16b5bda02012 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:56.268838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb8f63b-c2df-414f-891b-16b5bda02012 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:56.276507 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb8f63b-c2df-414f-891b-16b5bda02012 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:56.277709 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb8f63b-c2df-414f-891b-16b5bda02012 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-5eb8f63b-c2df-414f-891b-16b5bda02012 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:55:56.290068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:56.296831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:56.298107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:56.298107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:56.298107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:55:56.298107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:55:56.298107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:55:56.298107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:55:56.298107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:55:56.298107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:55:56.298107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 31 00:55:56.298107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:56.298107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 557/1094] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Jul 31 00:55:56 2022] GET /identity/v3/users/nova => generated 82 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:56.298107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb8f63b-c2df-414f-891b-16b5bda02012 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:56.355324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d30a355-48db-4c16-b0ea-68ab6e26f627 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:56.390851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c946c90-cd75-45c7-a77e-845a796ff8ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:56.391498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c946c90-cd75-45c7-a77e-845a796ff8ac None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:56.391776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c946c90-cd75-45c7-a77e-845a796ff8ac None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:56.392049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c946c90-cd75-45c7-a77e-845a796ff8ac None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:56.392782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c946c90-cd75-45c7-a77e-845a796ff8ac None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:56.393084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c946c90-cd75-45c7-a77e-845a796ff8ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:56.394474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c946c90-cd75-45c7-a77e-845a796ff8ac None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:56.399785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb8f63b-c2df-414f-891b-16b5bda02012 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:56.403236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 538/1095] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:55:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:56.409679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c946c90-cd75-45c7-a77e-845a796ff8ac None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:56.421307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:56.436315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9525a1-4997-4c62-a080-5d3f98efa310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:56.436932 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9525a1-4997-4c62-a080-5d3f98efa310 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:56.437198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9525a1-4997-4c62-a080-5d3f98efa310 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:56.437473 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9525a1-4997-4c62-a080-5d3f98efa310 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:56.438263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9525a1-4997-4c62-a080-5d3f98efa310 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:56.438650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9525a1-4997-4c62-a080-5d3f98efa310 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:56.440025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9525a1-4997-4c62-a080-5d3f98efa310 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:56.447730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 558/1096] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Jul 31 00:55:56 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:56.464884 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9525a1-4997-4c62-a080-5d3f98efa310 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:56.515260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c946c90-cd75-45c7-a77e-845a796ff8ac None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:56.555844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766a9ded-b841-4985-a8bd-c7f190352b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:56.556509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766a9ded-b841-4985-a8bd-c7f190352b12 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:56.556786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766a9ded-b841-4985-a8bd-c7f190352b12 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:56.557057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766a9ded-b841-4985-a8bd-c7f190352b12 None admin] PATH_INFO: `/v3/users/1ab7192596894bd2bebd3a28316cec06` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:56.558219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766a9ded-b841-4985-a8bd-c7f190352b12 None admin] RBAC: Authorizing `identity:get_user(user_id=1ab7192596894bd2bebd3a28316cec06)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:56.559677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766a9ded-b841-4985-a8bd-c7f190352b12 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:56.561833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 539/1097] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:55:56 2022] GET /identity/v3/users/1ab7192596894bd2bebd3a28316cec06 => generated 285 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:56.567478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766a9ded-b841-4985-a8bd-c7f190352b12 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:56.580884 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9525a1-4997-4c62-a080-5d3f98efa310 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:56.623851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b50eb96-f088-4e73-ba81-15331cd56d82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:56.624458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b50eb96-f088-4e73-ba81-15331cd56d82 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:56.624805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b50eb96-f088-4e73-ba81-15331cd56d82 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:56.625162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b50eb96-f088-4e73-ba81-15331cd56d82 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:56.676166 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:56.679708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766a9ded-b841-4985-a8bd-c7f190352b12 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:56.682229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:56.700217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b50eb96-f088-4e73-ba81-15331cd56d82 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:56.702207 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b50eb96-f088-4e73-ba81-15331cd56d82 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:56.717145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af004c0-da8d-453b-b418-76b3d37d6772 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:56.717770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af004c0-da8d-453b-b418-76b3d37d6772 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:56.718051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af004c0-da8d-453b-b418-76b3d37d6772 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:56.718346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af004c0-da8d-453b-b418-76b3d37d6772 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:56.788646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af004c0-da8d-453b-b418-76b3d37d6772 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:56.790112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af004c0-da8d-453b-b418-76b3d37d6772 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:56.811407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 559/1098] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:56 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:55:56.817609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b50eb96-f088-4e73-ba81-15331cd56d82 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:56.915123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:56.926801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 540/1099] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:55:56 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 31 00:55:56.928012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af004c0-da8d-453b-b418-76b3d37d6772 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:56.932197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:56.943014 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b50eb96-f088-4e73-ba81-15331cd56d82 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:56.980217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7566a6-a39b-4d54-b61a-d11104c569bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:56.980864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7566a6-a39b-4d54-b61a-d11104c569bf None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:56.981150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7566a6-a39b-4d54-b61a-d11104c569bf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:56.981420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7566a6-a39b-4d54-b61a-d11104c569bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:56.992489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7566a6-a39b-4d54-b61a-d11104c569bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:56.993982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7566a6-a39b-4d54-b61a-d11104c569bf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-9f7566a6-a39b-4d54-b61a-d11104c569bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:55:57.010158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:57.011528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:57.012398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:55:57.012398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:55:57.012398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:55:57.012398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:57.012398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 560/1100] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:57.017769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7566a6-a39b-4d54-b61a-d11104c569bf None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:57.040540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af004c0-da8d-453b-b418-76b3d37d6772 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:57.081589 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1bf405-4cfe-4d6b-9fdd-611399c03f7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:57.082224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1bf405-4cfe-4d6b-9fdd-611399c03f7f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:57.082496 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1bf405-4cfe-4d6b-9fdd-611399c03f7f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:57.082761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1bf405-4cfe-4d6b-9fdd-611399c03f7f None admin] PATH_INFO: `/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:57.083922 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1bf405-4cfe-4d6b-9fdd-611399c03f7f None admin] RBAC: Authorizing `identity:get_project(project_id=df1ae0d2fe6d4e89b266674bfd40f1f4)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:57.085349 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1bf405-4cfe-4d6b-9fdd-611399c03f7f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:57.087011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 541/1101] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:55:56 2022] GET /identity/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4 => generated 311 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:57.092991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1bf405-4cfe-4d6b-9fdd-611399c03f7f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:57.131544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7566a6-a39b-4d54-b61a-d11104c569bf None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:57.164335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:57.172037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a07fc87-11d8-446a-ab5b-1755b7163ee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:57.172628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a07fc87-11d8-446a-ab5b-1755b7163ee2 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:57.172885 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a07fc87-11d8-446a-ab5b-1755b7163ee2 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:57.173166 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a07fc87-11d8-446a-ab5b-1755b7163ee2 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:57.173866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a07fc87-11d8-446a-ab5b-1755b7163ee2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:57.174171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a07fc87-11d8-446a-ab5b-1755b7163ee2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:57.175858 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a07fc87-11d8-446a-ab5b-1755b7163ee2 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:57.184502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:57.189481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 561/1102] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:55:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:57.195626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a07fc87-11d8-446a-ab5b-1755b7163ee2 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:57.218588 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1bf405-4cfe-4d6b-9fdd-611399c03f7f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:57.257395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3765db10-aa8e-479d-8332-c076441d66ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:57.258678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3765db10-aa8e-479d-8332-c076441d66ff None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:57.258678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3765db10-aa8e-479d-8332-c076441d66ff None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:57.258678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3765db10-aa8e-479d-8332-c076441d66ff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:57.259026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3765db10-aa8e-479d-8332-c076441d66ff None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=df1ae0d2fe6d4e89b266674bfd40f1f4, user.id=1ab7192596894bd2bebd3a28316cec06) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:57.259341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3765db10-aa8e-479d-8332-c076441d66ff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:57.260807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3765db10-aa8e-479d-8332-c076441d66ff None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:57.276326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 542/1103] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:57 2022] GET /identity/v3/role_assignments?user.id=1ab7192596894bd2bebd3a28316cec06&scope.project.id=df1ae0d2fe6d4e89b266674bfd40f1f4&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:55:57.316167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a07fc87-11d8-446a-ab5b-1755b7163ee2 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:57.371297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c671499-94c0-4885-bf46-fb2de913a825 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:57.371988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c671499-94c0-4885-bf46-fb2de913a825 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:57.372283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c671499-94c0-4885-bf46-fb2de913a825 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:57.373138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c671499-94c0-4885-bf46-fb2de913a825 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:57.380924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c671499-94c0-4885-bf46-fb2de913a825 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:57.382376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c671499-94c0-4885-bf46-fb2de913a825 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-8c671499-94c0-4885-bf46-fb2de913a825 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:55:57.394887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:55:57.396062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:55:57.397043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:55:57.397043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:55:57.397043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:55:57.397043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:55:57.397043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:55:57.397043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:55:57.397043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:55:57.397043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:55:57.397885 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 562/1104] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:55:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:55:57.402091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3765db10-aa8e-479d-8332-c076441d66ff None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:57.510237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3765db10-aa8e-479d-8332-c076441d66ff None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:57.548478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110f9624-9ed6-4440-8122-85204856cf65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:57.549219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110f9624-9ed6-4440-8122-85204856cf65 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:57.549367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110f9624-9ed6-4440-8122-85204856cf65 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:57.549633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110f9624-9ed6-4440-8122-85204856cf65 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:57.550624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110f9624-9ed6-4440-8122-85204856cf65 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:57.550923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110f9624-9ed6-4440-8122-85204856cf65 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:57.552371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110f9624-9ed6-4440-8122-85204856cf65 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:57.555004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:57.566093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 543/1105] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:55:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:57.571978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c671499-94c0-4885-bf46-fb2de913a825 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:57.632105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:57.684590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c671499-94c0-4885-bf46-fb2de913a825 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:57.718161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c37d3ae-0726-4cb1-92a6-3daf547ebe2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:57.718772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c37d3ae-0726-4cb1-92a6-3daf547ebe2c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:57.719046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c37d3ae-0726-4cb1-92a6-3daf547ebe2c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:57.719316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c37d3ae-0726-4cb1-92a6-3daf547ebe2c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:57.720120 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c37d3ae-0726-4cb1-92a6-3daf547ebe2c None admin] RBAC: Adding query filter params (role.id=53d929a69ffa476eac93c56d7de72a16, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=5b0bd572ae7a4c7f9e07fe0b9708aa7e) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:55:57.720425 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c37d3ae-0726-4cb1-92a6-3daf547ebe2c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:57.721845 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c37d3ae-0726-4cb1-92a6-3daf547ebe2c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:57.739385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 563/1106] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:55:57 2022] GET /identity/v3/role_assignments?user.id=5b0bd572ae7a4c7f9e07fe0b9708aa7e&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=53d929a69ffa476eac93c56d7de72a16 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:59.347942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4df203-e657-4c89-9e15-40f2f601c787 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:59.348513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4df203-e657-4c89-9e15-40f2f601c787 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:59.348513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4df203-e657-4c89-9e15-40f2f601c787 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:59.349963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 544/1107] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:55:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:55:59.354896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f560d43-4281-45dd-bad6-131d929fcdb1 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:59.355285 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f560d43-4281-45dd-bad6-131d929fcdb1 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:59.355513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f560d43-4281-45dd-bad6-131d929fcdb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:59.374888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-5f560d43-4281-45dd-bad6-131d929fcdb1 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:55:59.376114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5f560d43-4281-45dd-bad6-131d929fcdb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:55:59.440677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 564/1108] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:55:59 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:55:59.465831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4df203-e657-4c89-9e15-40f2f601c787 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:59.573244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4df203-e657-4c89-9e15-40f2f601c787 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:59.606004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b08ce23-4405-418a-a1dd-b064edafda55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:59.606550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b08ce23-4405-418a-a1dd-b064edafda55 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:59.607344 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b08ce23-4405-418a-a1dd-b064edafda55 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:59.607642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b08ce23-4405-418a-a1dd-b064edafda55 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:59.608776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b08ce23-4405-418a-a1dd-b064edafda55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:59.610198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b08ce23-4405-418a-a1dd-b064edafda55 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:59.611814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 545/1109] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:55:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:55:59.617607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f560d43-4281-45dd-bad6-131d929fcdb1 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:55:59.661811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:55:59.723503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f560d43-4281-45dd-bad6-131d929fcdb1 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:55:59.757067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c076b9be-11b7-4de3-88bc-03170ee5ddd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:55:59.757680 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c076b9be-11b7-4de3-88bc-03170ee5ddd9 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:55:59.757964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c076b9be-11b7-4de3-88bc-03170ee5ddd9 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:55:59.758237 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c076b9be-11b7-4de3-88bc-03170ee5ddd9 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:55:59.758915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c076b9be-11b7-4de3-88bc-03170ee5ddd9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:55:59.760366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c076b9be-11b7-4de3-88bc-03170ee5ddd9 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:55:59.781552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 565/1110] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:55:59 2022] POST /identity/v3/users => generated 286 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:56:01.839216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fcec8d-0b52-44cd-95e5-ae484fe275b8 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:01.839745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fcec8d-0b52-44cd-95e5-ae484fe275b8 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:01.839745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fcec8d-0b52-44cd-95e5-ae484fe275b8 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:01.841168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 546/1111] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:01.846485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0001f3-a618-45c8-80cb-7f3496e0e852 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:01.846897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0001f3-a618-45c8-80cb-7f3496e0e852 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:01.847032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0001f3-a618-45c8-80cb-7f3496e0e852 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:01.865281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ae0001f3-a618-45c8-80cb-7f3496e0e852 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:01.866453 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ae0001f3-a618-45c8-80cb-7f3496e0e852 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:01.932804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 566/1112] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:01 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:01.956327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fcec8d-0b52-44cd-95e5-ae484fe275b8 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:02.018168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:02.056094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fcec8d-0b52-44cd-95e5-ae484fe275b8 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:02.090070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6db2ee-7559-46d8-a9b1-31a60f02672f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:02.090615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6db2ee-7559-46d8-a9b1-31a60f02672f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:02.091430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6db2ee-7559-46d8-a9b1-31a60f02672f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:02.091690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6db2ee-7559-46d8-a9b1-31a60f02672f None admin] PATH_INFO: `/v3/roles/0a2f5ca94533412088499eaccd7b2e21` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:02.093096 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6db2ee-7559-46d8-a9b1-31a60f02672f None admin] RBAC: Authorizing `identity:get_role(role_id=0a2f5ca94533412088499eaccd7b2e21)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:02.094442 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6db2ee-7559-46d8-a9b1-31a60f02672f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:02.096176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 547/1113] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:56:01 2022] GET /identity/v3/roles/0a2f5ca94533412088499eaccd7b2e21 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:02.102668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0001f3-a618-45c8-80cb-7f3496e0e852 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:02.133859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:02.202446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0001f3-a618-45c8-80cb-7f3496e0e852 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:02.235337 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2f5701-9156-4a95-b5df-294a6840bc6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:02.235979 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2f5701-9156-4a95-b5df-294a6840bc6e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:02.236253 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2f5701-9156-4a95-b5df-294a6840bc6e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:02.236521 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2f5701-9156-4a95-b5df-294a6840bc6e None admin] PATH_INFO: `/v3/users/617e46e61c1b444eaf4a1e184a76fd7b` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:02.249315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2f5701-9156-4a95-b5df-294a6840bc6e None admin] RBAC: Authorizing `identity:get_user(user_id=617e46e61c1b444eaf4a1e184a76fd7b)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:02.250731 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2f5701-9156-4a95-b5df-294a6840bc6e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:02.252375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 567/1114] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:56:02 2022] GET /identity/v3/users/617e46e61c1b444eaf4a1e184a76fd7b => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:02.258193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6db2ee-7559-46d8-a9b1-31a60f02672f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:02.354334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6db2ee-7559-46d8-a9b1-31a60f02672f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:02.382653 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:02.388068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226f16ee-1c6b-4d60-94d1-1d4d4726a73e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:02.389015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226f16ee-1c6b-4d60-94d1-1d4d4726a73e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:02.389418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226f16ee-1c6b-4d60-94d1-1d4d4726a73e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:02.389814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226f16ee-1c6b-4d60-94d1-1d4d4726a73e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:02.457324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226f16ee-1c6b-4d60-94d1-1d4d4726a73e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:02.458750 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226f16ee-1c6b-4d60-94d1-1d4d4726a73e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:02.562981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 548/1115] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:56:02 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:56:02.568859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2f5701-9156-4a95-b5df-294a6840bc6e None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:02.657096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:02.669765 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2f5701-9156-4a95-b5df-294a6840bc6e None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:02.703407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db84072-57c1-47f7-ac37-7f1dd5f01573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:02.704041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db84072-57c1-47f7-ac37-7f1dd5f01573 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:02.704326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db84072-57c1-47f7-ac37-7f1dd5f01573 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:02.704597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db84072-57c1-47f7-ac37-7f1dd5f01573 None admin] PATH_INFO: `/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:02.705741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db84072-57c1-47f7-ac37-7f1dd5f01573 None admin] RBAC: Authorizing `identity:get_project(project_id=df1ae0d2fe6d4e89b266674bfd40f1f4)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:02.707165 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db84072-57c1-47f7-ac37-7f1dd5f01573 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:02.708857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 568/1116] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:56:02 2022] GET /identity/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:02.715200 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226f16ee-1c6b-4d60-94d1-1d4d4726a73e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:02.744921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:02.811989 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226f16ee-1c6b-4d60-94d1-1d4d4726a73e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:02.844824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2e6f05-a31d-481c-a2a5-cb5f1cb10651 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:02.845462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2e6f05-a31d-481c-a2a5-cb5f1cb10651 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:02.845684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2e6f05-a31d-481c-a2a5-cb5f1cb10651 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:02.845952 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2e6f05-a31d-481c-a2a5-cb5f1cb10651 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:02.846712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2e6f05-a31d-481c-a2a5-cb5f1cb10651 None admin] RBAC: Adding query filter params (role.id=0a2f5ca94533412088499eaccd7b2e21, scope.project.id=df1ae0d2fe6d4e89b266674bfd40f1f4, user.id=617e46e61c1b444eaf4a1e184a76fd7b) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:02.847016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2e6f05-a31d-481c-a2a5-cb5f1cb10651 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:02.848468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2e6f05-a31d-481c-a2a5-cb5f1cb10651 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:02.853920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 549/1117] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:56:02 2022] GET /identity/v3/role_assignments?user.id=617e46e61c1b444eaf4a1e184a76fd7b&scope.project.id=df1ae0d2fe6d4e89b266674bfd40f1f4&role.id=0a2f5ca94533412088499eaccd7b2e21 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:04.861014 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ad5234-e8cd-4875-ac79-7b542e6b9475 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:04.861533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ad5234-e8cd-4875-ac79-7b542e6b9475 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:04.861533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ad5234-e8cd-4875-ac79-7b542e6b9475 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:04.862821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 569/1118] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:04.868452 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f39322-26ef-47c8-b780-300b8b760ec3 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:04.868891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f39322-26ef-47c8-b780-300b8b760ec3 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:04.868997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f39322-26ef-47c8-b780-300b8b760ec3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:04.886842 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-90f39322-26ef-47c8-b780-300b8b760ec3 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:04.889046 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-90f39322-26ef-47c8-b780-300b8b760ec3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:04.956390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 550/1119] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:04 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:04.979392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ad5234-e8cd-4875-ac79-7b542e6b9475 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:05.081020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ad5234-e8cd-4875-ac79-7b542e6b9475 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:05.118189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea1c4af-d8fe-412e-bff8-668e63a30ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:05.118728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea1c4af-d8fe-412e-bff8-668e63a30ab4 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:05.119006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea1c4af-d8fe-412e-bff8-668e63a30ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:05.119277 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea1c4af-d8fe-412e-bff8-668e63a30ab4 None admin] PATH_INFO: `/v3/roles/0a2f5ca94533412088499eaccd7b2e21` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:05.120783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea1c4af-d8fe-412e-bff8-668e63a30ab4 None admin] RBAC: Authorizing `identity:get_role(role_id=0a2f5ca94533412088499eaccd7b2e21)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:05.122692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea1c4af-d8fe-412e-bff8-668e63a30ab4 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:05.124114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 570/1120] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:56:04 2022] GET /identity/v3/roles/0a2f5ca94533412088499eaccd7b2e21 => generated 223 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:05.131052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f39322-26ef-47c8-b780-300b8b760ec3 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:05.160506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:05.244882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f39322-26ef-47c8-b780-300b8b760ec3 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:05.281454 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662273b6-bac7-459d-9a09-94e6ff44339f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:05.282095 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662273b6-bac7-459d-9a09-94e6ff44339f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:05.282377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662273b6-bac7-459d-9a09-94e6ff44339f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:05.282651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662273b6-bac7-459d-9a09-94e6ff44339f None admin] PATH_INFO: `/v3/users/617e46e61c1b444eaf4a1e184a76fd7b` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:05.283831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662273b6-bac7-459d-9a09-94e6ff44339f None admin] RBAC: Authorizing `identity:get_user(user_id=617e46e61c1b444eaf4a1e184a76fd7b)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:05.285271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662273b6-bac7-459d-9a09-94e6ff44339f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:05.286895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 551/1121] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:56:05 2022] GET /identity/v3/users/617e46e61c1b444eaf4a1e184a76fd7b => generated 286 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:56:05.292859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea1c4af-d8fe-412e-bff8-668e63a30ab4 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:05.319298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:05.397259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea1c4af-d8fe-412e-bff8-668e63a30ab4 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:05.432446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d886dd5f-f982-4bdb-9fe9-eee411dad0ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:05.433093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d886dd5f-f982-4bdb-9fe9-eee411dad0ea None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:05.433372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d886dd5f-f982-4bdb-9fe9-eee411dad0ea None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:05.433642 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d886dd5f-f982-4bdb-9fe9-eee411dad0ea None admin] PATH_INFO: `/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:05.435306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d886dd5f-f982-4bdb-9fe9-eee411dad0ea None admin] RBAC: Authorizing `identity:get_project(project_id=df1ae0d2fe6d4e89b266674bfd40f1f4)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:05.436839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d886dd5f-f982-4bdb-9fe9-eee411dad0ea None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:05.438559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 571/1122] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:56:05 2022] GET /identity/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:05.444527 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662273b6-bac7-459d-9a09-94e6ff44339f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:05.540225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:05.548827 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662273b6-bac7-459d-9a09-94e6ff44339f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:05.583643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28668f27-84d3-4e1a-8f56-35bc028bbf2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:05.584311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28668f27-84d3-4e1a-8f56-35bc028bbf2d None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:05.584585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28668f27-84d3-4e1a-8f56-35bc028bbf2d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:05.584850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28668f27-84d3-4e1a-8f56-35bc028bbf2d None admin] PATH_INFO: `/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4/users/617e46e61c1b444eaf4a1e184a76fd7b/roles/0a2f5ca94533412088499eaccd7b2e21` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:05.587026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28668f27-84d3-4e1a-8f56-35bc028bbf2d None admin] RBAC: Authorizing `identity:create_grant(project_id=df1ae0d2fe6d4e89b266674bfd40f1f4, user_id=617e46e61c1b444eaf4a1e184a76fd7b, role_id=0a2f5ca94533412088499eaccd7b2e21)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:05.588485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28668f27-84d3-4e1a-8f56-35bc028bbf2d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:05.602712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 552/1123] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:56:05 2022] PUT /identity/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4/users/617e46e61c1b444eaf4a1e184a76fd7b/roles/0a2f5ca94533412088499eaccd7b2e21 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:56:07.635807 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8fa79a-786c-4733-8528-391bc68183df None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:07.636340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8fa79a-786c-4733-8528-391bc68183df None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:07.636340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8fa79a-786c-4733-8528-391bc68183df None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:07.638260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 572/1124] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:07.643708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7948835e-5d6d-4e7d-a5a4-6eb0d6ca2afa None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:07.644138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7948835e-5d6d-4e7d-a5a4-6eb0d6ca2afa None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:07.644262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7948835e-5d6d-4e7d-a5a4-6eb0d6ca2afa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:07.663581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-7948835e-5d6d-4e7d-a5a4-6eb0d6ca2afa None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:07.664820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7948835e-5d6d-4e7d-a5a4-6eb0d6ca2afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:07.739305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 553/1125] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:07 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:07.762972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8fa79a-786c-4733-8528-391bc68183df None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:07.859313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:07.869476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8fa79a-786c-4733-8528-391bc68183df None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:07.924149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebd64f0-d000-48d4-b5de-e1796521aaff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:07.925499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebd64f0-d000-48d4-b5de-e1796521aaff None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:07.926000 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebd64f0-d000-48d4-b5de-e1796521aaff None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:07.926495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebd64f0-d000-48d4-b5de-e1796521aaff None admin] PATH_INFO: `/v3/roles/0a2f5ca94533412088499eaccd7b2e21` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:07.929058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebd64f0-d000-48d4-b5de-e1796521aaff None admin] RBAC: Authorizing `identity:get_role(role_id=0a2f5ca94533412088499eaccd7b2e21)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:07.931705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebd64f0-d000-48d4-b5de-e1796521aaff None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:07.934290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 573/1126] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:56:07 2022] GET /identity/v3/roles/0a2f5ca94533412088499eaccd7b2e21 => generated 223 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:07.940999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7948835e-5d6d-4e7d-a5a4-6eb0d6ca2afa None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:08.023384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:08.040308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7948835e-5d6d-4e7d-a5a4-6eb0d6ca2afa None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:08.076293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f19950-3987-4108-b14d-a8c754d10a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:08.076941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f19950-3987-4108-b14d-a8c754d10a61 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:08.077221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f19950-3987-4108-b14d-a8c754d10a61 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:08.077494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f19950-3987-4108-b14d-a8c754d10a61 None admin] PATH_INFO: `/v3/users/617e46e61c1b444eaf4a1e184a76fd7b` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:08.078710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f19950-3987-4108-b14d-a8c754d10a61 None admin] RBAC: Authorizing `identity:get_user(user_id=617e46e61c1b444eaf4a1e184a76fd7b)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:08.080193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f19950-3987-4108-b14d-a8c754d10a61 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:08.081852 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 554/1127] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:56:07 2022] GET /identity/v3/users/617e46e61c1b444eaf4a1e184a76fd7b => generated 286 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:08.087774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebd64f0-d000-48d4-b5de-e1796521aaff None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:08.211992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebd64f0-d000-48d4-b5de-e1796521aaff None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:08.252553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184d371b-a8d9-48b8-8939-9a970c3734bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:08.253174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184d371b-a8d9-48b8-8939-9a970c3734bc None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:08.253454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184d371b-a8d9-48b8-8939-9a970c3734bc None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:08.253729 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184d371b-a8d9-48b8-8939-9a970c3734bc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:08.284415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:08.329134 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184d371b-a8d9-48b8-8939-9a970c3734bc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:08.330604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184d371b-a8d9-48b8-8939-9a970c3734bc None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:08.435087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 574/1128] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:56:08 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:56:08.441325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f19950-3987-4108-b14d-a8c754d10a61 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:08.547695 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f19950-3987-4108-b14d-a8c754d10a61 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:08.582741 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73616aae-b14b-4dd7-8fb9-784bd3d348f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:08.583398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73616aae-b14b-4dd7-8fb9-784bd3d348f2 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:08.583689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73616aae-b14b-4dd7-8fb9-784bd3d348f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:08.583954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73616aae-b14b-4dd7-8fb9-784bd3d348f2 None admin] PATH_INFO: `/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:08.585091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73616aae-b14b-4dd7-8fb9-784bd3d348f2 None admin] RBAC: Authorizing `identity:get_project(project_id=df1ae0d2fe6d4e89b266674bfd40f1f4)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:08.586526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73616aae-b14b-4dd7-8fb9-784bd3d348f2 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:08.588248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 555/1129] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:56:08 2022] GET /identity/v3/projects/df1ae0d2fe6d4e89b266674bfd40f1f4 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:08.594889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184d371b-a8d9-48b8-8939-9a970c3734bc None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:08.667584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:08.710135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184d371b-a8d9-48b8-8939-9a970c3734bc None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:08.744496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bcabd3-9a7b-4489-ab3a-e1f9f5cd485f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:08.745103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bcabd3-9a7b-4489-ab3a-e1f9f5cd485f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:08.745375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bcabd3-9a7b-4489-ab3a-e1f9f5cd485f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:08.745645 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bcabd3-9a7b-4489-ab3a-e1f9f5cd485f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:08.746427 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bcabd3-9a7b-4489-ab3a-e1f9f5cd485f None admin] RBAC: Adding query filter params (role.id=0a2f5ca94533412088499eaccd7b2e21, scope.project.id=df1ae0d2fe6d4e89b266674bfd40f1f4, user.id=617e46e61c1b444eaf4a1e184a76fd7b) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:08.746730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bcabd3-9a7b-4489-ab3a-e1f9f5cd485f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:08.748808 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bcabd3-9a7b-4489-ab3a-e1f9f5cd485f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:08.755176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 575/1130] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:56:08 2022] GET /identity/v3/role_assignments?user.id=617e46e61c1b444eaf4a1e184a76fd7b&scope.project.id=df1ae0d2fe6d4e89b266674bfd40f1f4&role.id=0a2f5ca94533412088499eaccd7b2e21 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:10.862322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b200de-3370-470d-9135-24d00f6295d7 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:10.862827 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b200de-3370-470d-9135-24d00f6295d7 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:10.862827 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b200de-3370-470d-9135-24d00f6295d7 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:10.864240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 556/1131] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:10.869451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973037d8-4fcc-4a79-b476-60d02635a944 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:10.869900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973037d8-4fcc-4a79-b476-60d02635a944 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:10.870001 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973037d8-4fcc-4a79-b476-60d02635a944 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:10.889121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-973037d8-4fcc-4a79-b476-60d02635a944 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:10.890308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-973037d8-4fcc-4a79-b476-60d02635a944 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:10.961771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 576/1132] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:10 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:10.985294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b200de-3370-470d-9135-24d00f6295d7 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:11.007425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:11.089508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b200de-3370-470d-9135-24d00f6295d7 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:11.124636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445138e7-bd56-47ce-b20b-373cf17dab05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:11.125192 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445138e7-bd56-47ce-b20b-373cf17dab05 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:11.125471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445138e7-bd56-47ce-b20b-373cf17dab05 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:11.125748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445138e7-bd56-47ce-b20b-373cf17dab05 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:11.127186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445138e7-bd56-47ce-b20b-373cf17dab05 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:11.128828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445138e7-bd56-47ce-b20b-373cf17dab05 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:11.130678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 557/1133] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:56:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:11.136966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973037d8-4fcc-4a79-b476-60d02635a944 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:11.192512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:11.242161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973037d8-4fcc-4a79-b476-60d02635a944 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:11.278609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150c6a6f-2f38-430c-8724-59966fa97240 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:11.279232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150c6a6f-2f38-430c-8724-59966fa97240 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:11.279547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150c6a6f-2f38-430c-8724-59966fa97240 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:11.279820 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150c6a6f-2f38-430c-8724-59966fa97240 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:11.280619 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150c6a6f-2f38-430c-8724-59966fa97240 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:11.281985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150c6a6f-2f38-430c-8724-59966fa97240 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:11.311181 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 577/1134] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:56:11 2022] POST /identity/v3/projects => generated 311 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:56:13.389207 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df099c9b-7a0b-4afd-ad97-7568789631e3 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:13.389810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df099c9b-7a0b-4afd-ad97-7568789631e3 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:13.389810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df099c9b-7a0b-4afd-ad97-7568789631e3 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:13.391100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 558/1135] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:13.397154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ef4653-5ed0-4dd9-8279-9c2f07252c0a None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:13.397673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ef4653-5ed0-4dd9-8279-9c2f07252c0a None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:13.397762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ef4653-5ed0-4dd9-8279-9c2f07252c0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:13.419136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-79ef4653-5ed0-4dd9-8279-9c2f07252c0a None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:13.420381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-79ef4653-5ed0-4dd9-8279-9c2f07252c0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:13.486945 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 578/1136] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:13 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:13.511273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df099c9b-7a0b-4afd-ad97-7568789631e3 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:13.610540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:13.623653 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df099c9b-7a0b-4afd-ad97-7568789631e3 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:13.659189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ed2e00-363e-4dad-a2a2-df94911c4647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:13.659857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ed2e00-363e-4dad-a2a2-df94911c4647 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:13.660137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ed2e00-363e-4dad-a2a2-df94911c4647 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:13.660407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ed2e00-363e-4dad-a2a2-df94911c4647 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:13.661553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ed2e00-363e-4dad-a2a2-df94911c4647 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:13.662983 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ed2e00-363e-4dad-a2a2-df94911c4647 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:13.664615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 559/1137] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:56:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:13.671059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ef4653-5ed0-4dd9-8279-9c2f07252c0a None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:13.705296 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:13.781106 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ef4653-5ed0-4dd9-8279-9c2f07252c0a None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:13.818737 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64db0981-c29d-4fe6-99a6-171e14aacd4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:13.819404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64db0981-c29d-4fe6-99a6-171e14aacd4a None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:13.819685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64db0981-c29d-4fe6-99a6-171e14aacd4a None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:13.819958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64db0981-c29d-4fe6-99a6-171e14aacd4a None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:13.820663 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64db0981-c29d-4fe6-99a6-171e14aacd4a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:13.822105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64db0981-c29d-4fe6-99a6-171e14aacd4a None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:13.845743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 579/1138] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:56:13 2022] POST /identity/v3/users => generated 286 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:56:15.951514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7669fe4-28e9-4428-a60b-574ea9456549 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:15.952046 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7669fe4-28e9-4428-a60b-574ea9456549 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:15.952046 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7669fe4-28e9-4428-a60b-574ea9456549 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:15.953391 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 560/1139] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:15.958821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406abc66-ca86-4ba7-8787-4dced55105ce None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:15.959270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406abc66-ca86-4ba7-8787-4dced55105ce None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:15.959426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406abc66-ca86-4ba7-8787-4dced55105ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:15.977234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-406abc66-ca86-4ba7-8787-4dced55105ce None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:15.978413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-406abc66-ca86-4ba7-8787-4dced55105ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:16.051335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 580/1140] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:15 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:56:16.075492 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7669fe4-28e9-4428-a60b-574ea9456549 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:16.196456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7669fe4-28e9-4428-a60b-574ea9456549 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:16.232045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910d2071-a16c-4ef9-9ff5-cafdc9c4dc52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:16.232607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910d2071-a16c-4ef9-9ff5-cafdc9c4dc52 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:16.232883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910d2071-a16c-4ef9-9ff5-cafdc9c4dc52 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:16.233154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910d2071-a16c-4ef9-9ff5-cafdc9c4dc52 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:16.244304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910d2071-a16c-4ef9-9ff5-cafdc9c4dc52 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-910d2071-a16c-4ef9-9ff5-cafdc9c4dc52 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:56:16.250092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:56:16.251233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:56:16.252122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 561/1141] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:56:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:56:16.259498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406abc66-ca86-4ba7-8787-4dced55105ce None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:16.285811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:16.368766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406abc66-ca86-4ba7-8787-4dced55105ce None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:16.408045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496028c-855b-4401-bea5-7ae05c94cd80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:16.408658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496028c-855b-4401-bea5-7ae05c94cd80 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:16.408930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496028c-855b-4401-bea5-7ae05c94cd80 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:16.409196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496028c-855b-4401-bea5-7ae05c94cd80 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:16.409917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2496028c-855b-4401-bea5-7ae05c94cd80 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:16.410217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2496028c-855b-4401-bea5-7ae05c94cd80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:16.411631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2496028c-855b-4401-bea5-7ae05c94cd80 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:16.420851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 581/1142] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:56:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:16.427278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910d2071-a16c-4ef9-9ff5-cafdc9c4dc52 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:16.447435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:16.547393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910d2071-a16c-4ef9-9ff5-cafdc9c4dc52 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:16.586736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c49632-b121-4645-b9d4-dbfaafe4322f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:16.587406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c49632-b121-4645-b9d4-dbfaafe4322f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:16.587690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c49632-b121-4645-b9d4-dbfaafe4322f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:16.587970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c49632-b121-4645-b9d4-dbfaafe4322f None admin] PATH_INFO: `/v3/users/f365725d99294d6dab678e1c7dada233` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:16.602851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c49632-b121-4645-b9d4-dbfaafe4322f None admin] RBAC: Authorizing `identity:get_user(user_id=f365725d99294d6dab678e1c7dada233)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:16.604361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c49632-b121-4645-b9d4-dbfaafe4322f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:16.606141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 562/1143] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:56:16 2022] GET /identity/v3/users/f365725d99294d6dab678e1c7dada233 => generated 286 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:56:16.612533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496028c-855b-4401-bea5-7ae05c94cd80 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:16.709426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:16.726213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496028c-855b-4401-bea5-7ae05c94cd80 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:16.764241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89ae8de-1d3c-4d43-b3cc-8abf4764f44b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:16.764852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89ae8de-1d3c-4d43-b3cc-8abf4764f44b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:16.765127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89ae8de-1d3c-4d43-b3cc-8abf4764f44b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:16.765393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89ae8de-1d3c-4d43-b3cc-8abf4764f44b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:16.848496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89ae8de-1d3c-4d43-b3cc-8abf4764f44b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:16.849897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89ae8de-1d3c-4d43-b3cc-8abf4764f44b None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:16.942142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:16.958827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 582/1144] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:56:16 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 31 00:56:16.964823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c49632-b121-4645-b9d4-dbfaafe4322f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:17.038733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:17.064461 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c49632-b121-4645-b9d4-dbfaafe4322f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:17.097934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8bac7b-1d9d-4d07-a184-4059a57ba7dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:17.098617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8bac7b-1d9d-4d07-a184-4059a57ba7dd None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:17.098839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8bac7b-1d9d-4d07-a184-4059a57ba7dd None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:17.099106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8bac7b-1d9d-4d07-a184-4059a57ba7dd None admin] PATH_INFO: `/v3/projects/b3a04a75e8d341d6aa0493b3260c334b` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:17.100250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8bac7b-1d9d-4d07-a184-4059a57ba7dd None admin] RBAC: Authorizing `identity:get_project(project_id=b3a04a75e8d341d6aa0493b3260c334b)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:17.101665 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8bac7b-1d9d-4d07-a184-4059a57ba7dd None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:17.103317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 563/1145] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:56:16 2022] GET /identity/v3/projects/b3a04a75e8d341d6aa0493b3260c334b => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:17.110222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89ae8de-1d3c-4d43-b3cc-8abf4764f44b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:17.214505 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89ae8de-1d3c-4d43-b3cc-8abf4764f44b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:17.257094 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7057cdc6-25f1-4a3b-b2ba-cb25ae78a4bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:17.257708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7057cdc6-25f1-4a3b-b2ba-cb25ae78a4bf None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:17.257980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7057cdc6-25f1-4a3b-b2ba-cb25ae78a4bf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:17.258248 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7057cdc6-25f1-4a3b-b2ba-cb25ae78a4bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:17.259029 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7057cdc6-25f1-4a3b-b2ba-cb25ae78a4bf None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=b3a04a75e8d341d6aa0493b3260c334b, user.id=f365725d99294d6dab678e1c7dada233) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:17.259335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7057cdc6-25f1-4a3b-b2ba-cb25ae78a4bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:17.260788 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7057cdc6-25f1-4a3b-b2ba-cb25ae78a4bf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:17.266853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 583/1146] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:56:17 2022] GET /identity/v3/role_assignments?user.id=f365725d99294d6dab678e1c7dada233&scope.project.id=b3a04a75e8d341d6aa0493b3260c334b&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:19.265911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56902c4-81c0-4f47-a15c-b82667236f9f None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:19.266466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56902c4-81c0-4f47-a15c-b82667236f9f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:19.266466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56902c4-81c0-4f47-a15c-b82667236f9f None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:19.267863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 564/1147] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:19.273504 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006e6244-4c7b-4b2d-b309-a7cf2ee37b6f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:19.273986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006e6244-4c7b-4b2d-b309-a7cf2ee37b6f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:19.274076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006e6244-4c7b-4b2d-b309-a7cf2ee37b6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:19.292502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-006e6244-4c7b-4b2d-b309-a7cf2ee37b6f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:19.294219 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-006e6244-4c7b-4b2d-b309-a7cf2ee37b6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:19.341454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:19.369180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 584/1148] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:19 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:19.391929 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56902c4-81c0-4f47-a15c-b82667236f9f None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:19.507073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56902c4-81c0-4f47-a15c-b82667236f9f None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:19.542302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b99ac3-c567-4a01-a8c9-bd0390b81327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:19.542844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b99ac3-c567-4a01-a8c9-bd0390b81327 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:19.543173 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b99ac3-c567-4a01-a8c9-bd0390b81327 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:19.544393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b99ac3-c567-4a01-a8c9-bd0390b81327 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:19.555043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b99ac3-c567-4a01-a8c9-bd0390b81327 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-59b99ac3-c567-4a01-a8c9-bd0390b81327 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:56:19.559973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:56:19.561127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:56:19.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 565/1149] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:56:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:56:19.567557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006e6244-4c7b-4b2d-b309-a7cf2ee37b6f None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:19.665144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006e6244-4c7b-4b2d-b309-a7cf2ee37b6f None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:19.699927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9b31bc-c864-4f9c-aefc-c4db60cb29fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:19.700558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9b31bc-c864-4f9c-aefc-c4db60cb29fe None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:19.700822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9b31bc-c864-4f9c-aefc-c4db60cb29fe None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:19.701097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9b31bc-c864-4f9c-aefc-c4db60cb29fe None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:19.701806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9b31bc-c864-4f9c-aefc-c4db60cb29fe None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:19.702111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9b31bc-c864-4f9c-aefc-c4db60cb29fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:19.703513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9b31bc-c864-4f9c-aefc-c4db60cb29fe None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:19.711578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 585/1150] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:56:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:19.717529 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b99ac3-c567-4a01-a8c9-bd0390b81327 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:19.721037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:19.825497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b99ac3-c567-4a01-a8c9-bd0390b81327 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:19.859990 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19aabd6-ee79-4a8f-b55c-4a73714eef31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:19.860622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19aabd6-ee79-4a8f-b55c-4a73714eef31 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:19.860899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19aabd6-ee79-4a8f-b55c-4a73714eef31 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:19.861171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19aabd6-ee79-4a8f-b55c-4a73714eef31 None admin] PATH_INFO: `/v3/users/f365725d99294d6dab678e1c7dada233` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:19.862310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19aabd6-ee79-4a8f-b55c-4a73714eef31 None admin] RBAC: Authorizing `identity:get_user(user_id=f365725d99294d6dab678e1c7dada233)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:19.863824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19aabd6-ee79-4a8f-b55c-4a73714eef31 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:19.865477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 566/1151] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:56:19 2022] GET /identity/v3/users/f365725d99294d6dab678e1c7dada233 => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:19.871330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9b31bc-c864-4f9c-aefc-c4db60cb29fe None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:19.928345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:19.992988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9b31bc-c864-4f9c-aefc-c4db60cb29fe None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:20.026135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b90604e-04e5-47ee-b72d-21929ddfe4b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:20.026797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b90604e-04e5-47ee-b72d-21929ddfe4b5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:20.027028 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b90604e-04e5-47ee-b72d-21929ddfe4b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:20.027297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b90604e-04e5-47ee-b72d-21929ddfe4b5 None admin] PATH_INFO: `/v3/projects/b3a04a75e8d341d6aa0493b3260c334b` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:20.029050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b90604e-04e5-47ee-b72d-21929ddfe4b5 None admin] RBAC: Authorizing `identity:get_project(project_id=b3a04a75e8d341d6aa0493b3260c334b)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:20.030491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b90604e-04e5-47ee-b72d-21929ddfe4b5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:20.032181 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 586/1152] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:56:19 2022] GET /identity/v3/projects/b3a04a75e8d341d6aa0493b3260c334b => generated 311 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:20.038341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19aabd6-ee79-4a8f-b55c-4a73714eef31 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:20.107129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:20.140909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19aabd6-ee79-4a8f-b55c-4a73714eef31 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:20.179141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee21983-ee0e-486a-bb40-d3680af4e30d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:20.179844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee21983-ee0e-486a-bb40-d3680af4e30d None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:20.180121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee21983-ee0e-486a-bb40-d3680af4e30d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:20.180393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee21983-ee0e-486a-bb40-d3680af4e30d None admin] PATH_INFO: `/v3/projects/b3a04a75e8d341d6aa0493b3260c334b/users/f365725d99294d6dab678e1c7dada233/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:20.182373 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee21983-ee0e-486a-bb40-d3680af4e30d None admin] RBAC: Authorizing `identity:create_grant(project_id=b3a04a75e8d341d6aa0493b3260c334b, user_id=f365725d99294d6dab678e1c7dada233, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:20.184365 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee21983-ee0e-486a-bb40-d3680af4e30d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:20.198737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 567/1153] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:56:20 2022] PUT /identity/v3/projects/b3a04a75e8d341d6aa0493b3260c334b/users/f365725d99294d6dab678e1c7dada233/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:56:22.227487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aed765c-e3f3-487b-85c6-ac9706db5982 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:22.228099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aed765c-e3f3-487b-85c6-ac9706db5982 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:22.228099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aed765c-e3f3-487b-85c6-ac9706db5982 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:22.229491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 587/1154] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:22.235950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f346666b-6c26-4c72-8d75-742bee941913 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:22.236473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f346666b-6c26-4c72-8d75-742bee941913 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:22.236579 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f346666b-6c26-4c72-8d75-742bee941913 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:22.256060 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-f346666b-6c26-4c72-8d75-742bee941913 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:22.257285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f346666b-6c26-4c72-8d75-742bee941913 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:22.333591 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 568/1155] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:22 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:22.358218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aed765c-e3f3-487b-85c6-ac9706db5982 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:22.475242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aed765c-e3f3-487b-85c6-ac9706db5982 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:22.485772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:22.514626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08b8430-6028-4470-92f5-aacd52dbfc52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:22.515271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08b8430-6028-4470-92f5-aacd52dbfc52 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:22.515582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08b8430-6028-4470-92f5-aacd52dbfc52 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:22.515854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08b8430-6028-4470-92f5-aacd52dbfc52 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:22.527517 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08b8430-6028-4470-92f5-aacd52dbfc52 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-d08b8430-6028-4470-92f5-aacd52dbfc52 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:56:22.532514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:56:22.534189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:56:22.536037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 588/1156] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:56:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:56:22.541548 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f346666b-6c26-4c72-8d75-742bee941913 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:22.610002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:22.652209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f346666b-6c26-4c72-8d75-742bee941913 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:22.688477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798caf3d-f045-482b-9538-828212f61d07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:22.689150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798caf3d-f045-482b-9538-828212f61d07 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:22.689470 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798caf3d-f045-482b-9538-828212f61d07 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:22.689795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798caf3d-f045-482b-9538-828212f61d07 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:22.690544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798caf3d-f045-482b-9538-828212f61d07 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:22.690885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798caf3d-f045-482b-9538-828212f61d07 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:22.692379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798caf3d-f045-482b-9538-828212f61d07 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:22.704003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 569/1157] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:56:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:56:22.714254 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08b8430-6028-4470-92f5-aacd52dbfc52 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:22.825979 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08b8430-6028-4470-92f5-aacd52dbfc52 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:22.860913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa25bd3-888a-451f-a0f2-609fa782e011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:22.861550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa25bd3-888a-451f-a0f2-609fa782e011 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:22.861825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa25bd3-888a-451f-a0f2-609fa782e011 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:22.862095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa25bd3-888a-451f-a0f2-609fa782e011 None admin] PATH_INFO: `/v3/users/f365725d99294d6dab678e1c7dada233` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:22.863270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa25bd3-888a-451f-a0f2-609fa782e011 None admin] RBAC: Authorizing `identity:get_user(user_id=f365725d99294d6dab678e1c7dada233)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:22.864736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa25bd3-888a-451f-a0f2-609fa782e011 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:22.866365 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 589/1158] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:56:22 2022] GET /identity/v3/users/f365725d99294d6dab678e1c7dada233 => generated 286 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:22.873084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798caf3d-f045-482b-9538-828212f61d07 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:22.981050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798caf3d-f045-482b-9538-828212f61d07 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:23.009218 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:23.028208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0e3f78-7d7a-44c6-ba6d-c2b970acc2f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:23.029253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0e3f78-7d7a-44c6-ba6d-c2b970acc2f9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:23.029739 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0e3f78-7d7a-44c6-ba6d-c2b970acc2f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:23.030215 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0e3f78-7d7a-44c6-ba6d-c2b970acc2f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:23.160554 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0e3f78-7d7a-44c6-ba6d-c2b970acc2f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:23.165119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0e3f78-7d7a-44c6-ba6d-c2b970acc2f9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:23.271300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 570/1159] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:56:22 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 31 00:56:23.277960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa25bd3-888a-451f-a0f2-609fa782e011 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:23.314767 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:23.384316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa25bd3-888a-451f-a0f2-609fa782e011 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:23.421453 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fdfb15-a23a-4907-a364-e7d793b1e572 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:23.422066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fdfb15-a23a-4907-a364-e7d793b1e572 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:23.422390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fdfb15-a23a-4907-a364-e7d793b1e572 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:23.422708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fdfb15-a23a-4907-a364-e7d793b1e572 None admin] PATH_INFO: `/v3/projects/b3a04a75e8d341d6aa0493b3260c334b` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:23.423970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fdfb15-a23a-4907-a364-e7d793b1e572 None admin] RBAC: Authorizing `identity:get_project(project_id=b3a04a75e8d341d6aa0493b3260c334b)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:23.425736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fdfb15-a23a-4907-a364-e7d793b1e572 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:23.427535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 590/1160] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:56:23 2022] GET /identity/v3/projects/b3a04a75e8d341d6aa0493b3260c334b => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:23.434119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0e3f78-7d7a-44c6-ba6d-c2b970acc2f9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:23.460434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:23.546075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0e3f78-7d7a-44c6-ba6d-c2b970acc2f9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:23.584089 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f6bc27-52b7-48fa-ad03-3923f60cf532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:23.584713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f6bc27-52b7-48fa-ad03-3923f60cf532 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:23.584994 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f6bc27-52b7-48fa-ad03-3923f60cf532 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:23.585267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f6bc27-52b7-48fa-ad03-3923f60cf532 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:23.586059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f6bc27-52b7-48fa-ad03-3923f60cf532 None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=b3a04a75e8d341d6aa0493b3260c334b, user.id=f365725d99294d6dab678e1c7dada233) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:23.586367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f6bc27-52b7-48fa-ad03-3923f60cf532 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:23.587850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f6bc27-52b7-48fa-ad03-3923f60cf532 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:23.594687 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 571/1161] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:56:23 2022] GET /identity/v3/role_assignments?user.id=f365725d99294d6dab678e1c7dada233&scope.project.id=b3a04a75e8d341d6aa0493b3260c334b&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:26.364553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb63336-e1c8-43ca-ba5f-840486c69f58 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:26.365103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb63336-e1c8-43ca-ba5f-840486c69f58 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:26.365103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb63336-e1c8-43ca-ba5f-840486c69f58 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:26.366426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 591/1162] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:26.372447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093f8249-a2df-4462-aebf-c92424dafc8e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:26.372923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093f8249-a2df-4462-aebf-c92424dafc8e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:26.373020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093f8249-a2df-4462-aebf-c92424dafc8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:26.393336 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-093f8249-a2df-4462-aebf-c92424dafc8e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:26.395437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-093f8249-a2df-4462-aebf-c92424dafc8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:26.514820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 572/1163] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:26 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:26.537118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb63336-e1c8-43ca-ba5f-840486c69f58 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:26.662655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb63336-e1c8-43ca-ba5f-840486c69f58 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:26.664997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:26.698891 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad6ad06-a315-408c-83f0-e82e2622b0cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:26.699581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad6ad06-a315-408c-83f0-e82e2622b0cd None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:26.699858 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad6ad06-a315-408c-83f0-e82e2622b0cd None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:26.700129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad6ad06-a315-408c-83f0-e82e2622b0cd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:26.770885 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad6ad06-a315-408c-83f0-e82e2622b0cd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:26.772435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad6ad06-a315-408c-83f0-e82e2622b0cd None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:26.880389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 592/1164] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:56:26 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:56:26.886621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093f8249-a2df-4462-aebf-c92424dafc8e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:26.904745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:26.997759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093f8249-a2df-4462-aebf-c92424dafc8e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:27.034789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db79d037-d35b-44e5-a1e9-be80c05eb426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:27.035466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db79d037-d35b-44e5-a1e9-be80c05eb426 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:27.035747 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db79d037-d35b-44e5-a1e9-be80c05eb426 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:27.036020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db79d037-d35b-44e5-a1e9-be80c05eb426 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:27.044308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db79d037-d35b-44e5-a1e9-be80c05eb426 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:27.045815 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db79d037-d35b-44e5-a1e9-be80c05eb426 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-db79d037-d35b-44e5-a1e9-be80c05eb426 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:56:27.058831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:56:27.060840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:56:27.061988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 573/1165] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Jul 31 00:56:26 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:56:27.066613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad6ad06-a315-408c-83f0-e82e2622b0cd None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:27.069899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:27.178190 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad6ad06-a315-408c-83f0-e82e2622b0cd None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:27.213254 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e06b7d-ae9b-4baf-a968-b90bc13feace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:27.213867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e06b7d-ae9b-4baf-a968-b90bc13feace None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:27.214143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e06b7d-ae9b-4baf-a968-b90bc13feace None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:27.215258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e06b7d-ae9b-4baf-a968-b90bc13feace None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:27.215973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e06b7d-ae9b-4baf-a968-b90bc13feace None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:27.216292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e06b7d-ae9b-4baf-a968-b90bc13feace None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:27.217624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e06b7d-ae9b-4baf-a968-b90bc13feace None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:27.225151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 593/1166] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Jul 31 00:56:27 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:56:29.271162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a073f22-da79-49da-a1cc-f17130a3a4e0 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:29.271772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a073f22-da79-49da-a1cc-f17130a3a4e0 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:29.271772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a073f22-da79-49da-a1cc-f17130a3a4e0 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:29.273091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 574/1167] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:29.279294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f91cbb4-1ff5-42ed-8964-f392c100b329 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:29.279857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f91cbb4-1ff5-42ed-8964-f392c100b329 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:29.279857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f91cbb4-1ff5-42ed-8964-f392c100b329 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:29.299874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-5f91cbb4-1ff5-42ed-8964-f392c100b329 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:29.301110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5f91cbb4-1ff5-42ed-8964-f392c100b329 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:29.370194 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:29.376502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 594/1168] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:29 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:29.399428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a073f22-da79-49da-a1cc-f17130a3a4e0 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:29.484128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:29.510189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a073f22-da79-49da-a1cc-f17130a3a4e0 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:29.548834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545e7c5a-9755-46b3-9f84-6f304f17295c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:29.549447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545e7c5a-9755-46b3-9f84-6f304f17295c None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:29.549738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545e7c5a-9755-46b3-9f84-6f304f17295c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:29.550013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545e7c5a-9755-46b3-9f84-6f304f17295c None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:29.550740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545e7c5a-9755-46b3-9f84-6f304f17295c None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:29.552150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545e7c5a-9755-46b3-9f84-6f304f17295c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:29.572095 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 575/1169] 10.222.0.42 () {60 vars in 1138 bytes} [Sun Jul 31 00:56:29 2022] POST /identity/v3/domains => generated 267 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:56:31.630067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e897b4cb-0ea7-481e-bae8-0ed4e5db36c6 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:31.630657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e897b4cb-0ea7-481e-bae8-0ed4e5db36c6 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:31.630657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e897b4cb-0ea7-481e-bae8-0ed4e5db36c6 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:31.632119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 595/1170] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:56:31.638512 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1ec780-a2b4-45ca-8415-7db2582f95a9 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:31.639078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1ec780-a2b4-45ca-8415-7db2582f95a9 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:31.639078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1ec780-a2b4-45ca-8415-7db2582f95a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:31.657899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-3d1ec780-a2b4-45ca-8415-7db2582f95a9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:31.659120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-3d1ec780-a2b4-45ca-8415-7db2582f95a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:31.729534 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 576/1171] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:31 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:31.753657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e897b4cb-0ea7-481e-bae8-0ed4e5db36c6 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:31.858062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e897b4cb-0ea7-481e-bae8-0ed4e5db36c6 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:31.894533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f13be7-91c1-4fa8-8a5e-cd9f4c018729 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:31.895183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f13be7-91c1-4fa8-8a5e-cd9f4c018729 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:31.895500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f13be7-91c1-4fa8-8a5e-cd9f4c018729 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:31.895771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f13be7-91c1-4fa8-8a5e-cd9f4c018729 None admin] PATH_INFO: `/v3/domains/51ff20dd6e29462c9b8d82af24cf6109` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:31.907441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f13be7-91c1-4fa8-8a5e-cd9f4c018729 None admin] RBAC: Authorizing `identity:get_domain(domain_id=51ff20dd6e29462c9b8d82af24cf6109)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:31.908914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f13be7-91c1-4fa8-8a5e-cd9f4c018729 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:31.910668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 596/1172] 10.222.0.42 () {56 vars in 1183 bytes} [Sun Jul 31 00:56:31 2022] GET /identity/v3/domains/51ff20dd6e29462c9b8d82af24cf6109 => generated 267 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:31.920872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1ec780-a2b4-45ca-8415-7db2582f95a9 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:31.988066 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:32.027498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1ec780-a2b4-45ca-8415-7db2582f95a9 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:32.065183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a44034-5c7d-446b-8bb4-e69ff283096b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:32.065791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a44034-5c7d-446b-8bb4-e69ff283096b None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:32.066061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a44034-5c7d-446b-8bb4-e69ff283096b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:32.066326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a44034-5c7d-446b-8bb4-e69ff283096b None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:32.067011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a44034-5c7d-446b-8bb4-e69ff283096b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:32.068445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a44034-5c7d-446b-8bb4-e69ff283096b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:32.097326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 577/1173] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:56:31 2022] POST /identity/v3/projects => generated 361 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 00:56:34.152287 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec839e3-5a12-48d5-83bf-39ee9e32804d None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:34.152816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec839e3-5a12-48d5-83bf-39ee9e32804d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:34.152816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec839e3-5a12-48d5-83bf-39ee9e32804d None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:34.154164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 597/1174] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:34.159504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea99c6a-2541-4814-8c86-2d9c6f381718 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:34.160198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea99c6a-2541-4814-8c86-2d9c6f381718 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:34.160427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea99c6a-2541-4814-8c86-2d9c6f381718 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:34.178650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-5ea99c6a-2541-4814-8c86-2d9c6f381718 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:34.179884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5ea99c6a-2541-4814-8c86-2d9c6f381718 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:34.250785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 578/1175] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:34 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:34.277459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec839e3-5a12-48d5-83bf-39ee9e32804d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:34.343183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:34.398584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec839e3-5a12-48d5-83bf-39ee9e32804d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:34.440167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4425e0-f7e8-4e2c-98ad-29e8030bce0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:34.440721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4425e0-f7e8-4e2c-98ad-29e8030bce0d None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:34.440992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4425e0-f7e8-4e2c-98ad-29e8030bce0d None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:34.441263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4425e0-f7e8-4e2c-98ad-29e8030bce0d None admin] PATH_INFO: `/v3/domains/51ff20dd6e29462c9b8d82af24cf6109` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:34.442450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4425e0-f7e8-4e2c-98ad-29e8030bce0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=51ff20dd6e29462c9b8d82af24cf6109)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:34.443920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4425e0-f7e8-4e2c-98ad-29e8030bce0d None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:34.445531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 598/1176] 10.222.0.42 () {56 vars in 1183 bytes} [Sun Jul 31 00:56:34 2022] GET /identity/v3/domains/51ff20dd6e29462c9b8d82af24cf6109 => generated 267 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:34.452065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea99c6a-2541-4814-8c86-2d9c6f381718 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:34.501541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:34.556186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea99c6a-2541-4814-8c86-2d9c6f381718 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:34.591366 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47df775c-af54-42d9-b230-e5e0c9267a4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:34.592010 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47df775c-af54-42d9-b230-e5e0c9267a4b None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:34.592290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47df775c-af54-42d9-b230-e5e0c9267a4b None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:34.592588 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47df775c-af54-42d9-b230-e5e0c9267a4b None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:34.593266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47df775c-af54-42d9-b230-e5e0c9267a4b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:34.594723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47df775c-af54-42d9-b230-e5e0c9267a4b None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:34.620917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 579/1177] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:56:34 2022] POST /identity/v3/users => generated 311 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:56:36.731500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244c624e-f929-4dad-a7dc-fff454ae791c None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:36.732056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244c624e-f929-4dad-a7dc-fff454ae791c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:36.732056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244c624e-f929-4dad-a7dc-fff454ae791c None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:36.733396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 599/1178] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:36.738961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c728e579-0c6f-4e6b-b6f1-c958b3d88b57 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:36.739460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c728e579-0c6f-4e6b-b6f1-c958b3d88b57 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:36.739557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c728e579-0c6f-4e6b-b6f1-c958b3d88b57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:36.757198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c728e579-0c6f-4e6b-b6f1-c958b3d88b57 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:36.758366 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c728e579-0c6f-4e6b-b6f1-c958b3d88b57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:36.824643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 580/1179] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:36 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:36.848516 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244c624e-f929-4dad-a7dc-fff454ae791c None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:36.955949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244c624e-f929-4dad-a7dc-fff454ae791c None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:36.983156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:36.995649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8ba7a5-9133-45ee-b136-24fb00bbee5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:36.996286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8ba7a5-9133-45ee-b136-24fb00bbee5b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:36.996563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8ba7a5-9133-45ee-b136-24fb00bbee5b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:36.996833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8ba7a5-9133-45ee-b136-24fb00bbee5b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:37.007108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8ba7a5-9133-45ee-b136-24fb00bbee5b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-7f8ba7a5-9133-45ee-b136-24fb00bbee5b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:56:37.012766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:56:37.013982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:56:37.015046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 600/1180] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:56:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:56:37.020447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c728e579-0c6f-4e6b-b6f1-c958b3d88b57 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:37.055111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:37.125334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c728e579-0c6f-4e6b-b6f1-c958b3d88b57 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:37.176452 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa33a89-d27a-4d60-a35f-29487854db43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:37.177070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa33a89-d27a-4d60-a35f-29487854db43 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:37.177346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa33a89-d27a-4d60-a35f-29487854db43 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:37.177617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa33a89-d27a-4d60-a35f-29487854db43 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:37.178321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa33a89-d27a-4d60-a35f-29487854db43 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:37.178620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa33a89-d27a-4d60-a35f-29487854db43 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:37.180016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa33a89-d27a-4d60-a35f-29487854db43 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:37.188933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 581/1181] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:56:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:37.194837 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8ba7a5-9133-45ee-b136-24fb00bbee5b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:37.291533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8ba7a5-9133-45ee-b136-24fb00bbee5b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:37.324893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40614cd-447b-44cd-aea0-f71e4d5e344d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:37.325524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40614cd-447b-44cd-aea0-f71e4d5e344d None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:37.325800 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40614cd-447b-44cd-aea0-f71e4d5e344d None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:37.326069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40614cd-447b-44cd-aea0-f71e4d5e344d None admin] PATH_INFO: `/v3/users/b562f81817e847deaa2210238e1bdd61` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:37.340005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40614cd-447b-44cd-aea0-f71e4d5e344d None admin] RBAC: Authorizing `identity:get_user(user_id=b562f81817e847deaa2210238e1bdd61)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:37.341446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40614cd-447b-44cd-aea0-f71e4d5e344d None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:37.343183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 601/1182] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:56:37 2022] GET /identity/v3/users/b562f81817e847deaa2210238e1bdd61 => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:37.349288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa33a89-d27a-4d60-a35f-29487854db43 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:37.444451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:37.450627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa33a89-d27a-4d60-a35f-29487854db43 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:37.484880 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592ba27c-cddf-45d5-aa35-7e0724677b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:37.485506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592ba27c-cddf-45d5-aa35-7e0724677b8e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:37.485786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592ba27c-cddf-45d5-aa35-7e0724677b8e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:37.486056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592ba27c-cddf-45d5-aa35-7e0724677b8e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:37.566873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592ba27c-cddf-45d5-aa35-7e0724677b8e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:37.568360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592ba27c-cddf-45d5-aa35-7e0724677b8e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:37.668148 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:37.675651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 582/1183] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:56:37 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:56:37.681684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40614cd-447b-44cd-aea0-f71e4d5e344d None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:37.736892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:37.801039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40614cd-447b-44cd-aea0-f71e4d5e344d None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:37.836530 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55908946-0aa6-42d9-afd9-c0cd56dd807a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:37.837190 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55908946-0aa6-42d9-afd9-c0cd56dd807a None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:37.837468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55908946-0aa6-42d9-afd9-c0cd56dd807a None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:37.837744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55908946-0aa6-42d9-afd9-c0cd56dd807a None admin] PATH_INFO: `/v3/projects/7587c8f5812e4d36a314a8526b242f41` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:37.838934 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55908946-0aa6-42d9-afd9-c0cd56dd807a None admin] RBAC: Authorizing `identity:get_project(project_id=7587c8f5812e4d36a314a8526b242f41)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:37.840402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55908946-0aa6-42d9-afd9-c0cd56dd807a None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:37.842103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 602/1184] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:56:37 2022] GET /identity/v3/projects/7587c8f5812e4d36a314a8526b242f41 => generated 361 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:37.851372 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592ba27c-cddf-45d5-aa35-7e0724677b8e None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:37.956141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592ba27c-cddf-45d5-aa35-7e0724677b8e None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:37.997050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b8afc4-364a-4531-bca9-b4c8e47dd11d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:37.997575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b8afc4-364a-4531-bca9-b4c8e47dd11d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:37.997846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b8afc4-364a-4531-bca9-b4c8e47dd11d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:37.998111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b8afc4-364a-4531-bca9-b4c8e47dd11d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:37.998892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b8afc4-364a-4531-bca9-b4c8e47dd11d None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=7587c8f5812e4d36a314a8526b242f41, user.id=b562f81817e847deaa2210238e1bdd61) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:37.999196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b8afc4-364a-4531-bca9-b4c8e47dd11d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:38.000651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b8afc4-364a-4531-bca9-b4c8e47dd11d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:38.006895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 583/1185] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:56:37 2022] GET /identity/v3/role_assignments?user.id=b562f81817e847deaa2210238e1bdd61&scope.project.id=7587c8f5812e4d36a314a8526b242f41&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:40.059689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfaddde-d094-4160-843a-fd43d7bd1ff9 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:40.060468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfaddde-d094-4160-843a-fd43d7bd1ff9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:40.060468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfaddde-d094-4160-843a-fd43d7bd1ff9 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:40.061663 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 603/1186] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:56:40.067758 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f8dbda-bbc9-4d60-8362-3dbef342a71c None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:40.068571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f8dbda-bbc9-4d60-8362-3dbef342a71c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:40.068571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f8dbda-bbc9-4d60-8362-3dbef342a71c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:40.088125 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d9f8dbda-bbc9-4d60-8362-3dbef342a71c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:40.089939 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d9f8dbda-bbc9-4d60-8362-3dbef342a71c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:40.156166 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:40.168379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 584/1187] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:40 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:56:40.191092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfaddde-d094-4160-843a-fd43d7bd1ff9 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:40.324462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfaddde-d094-4160-843a-fd43d7bd1ff9 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:40.345592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:40.369523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44230fc-806a-42b1-bbc9-f5727a197854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:40.370668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44230fc-806a-42b1-bbc9-f5727a197854 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:40.370944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44230fc-806a-42b1-bbc9-f5727a197854 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:40.371212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44230fc-806a-42b1-bbc9-f5727a197854 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:40.382097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44230fc-806a-42b1-bbc9-f5727a197854 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-a44230fc-806a-42b1-bbc9-f5727a197854 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:56:40.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:56:40.389360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:56:40.391215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 604/1188] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:56:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:56:40.396688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f8dbda-bbc9-4d60-8362-3dbef342a71c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:40.503165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f8dbda-bbc9-4d60-8362-3dbef342a71c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:40.542626 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f7a1f0-9444-442a-af68-27555a2c2a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:40.543251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f7a1f0-9444-442a-af68-27555a2c2a1d None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:40.543559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f7a1f0-9444-442a-af68-27555a2c2a1d None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:40.543831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f7a1f0-9444-442a-af68-27555a2c2a1d None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:40.544562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f7a1f0-9444-442a-af68-27555a2c2a1d None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:40.544856 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f7a1f0-9444-442a-af68-27555a2c2a1d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:40.546249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f7a1f0-9444-442a-af68-27555a2c2a1d None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:40.555765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 585/1189] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:56:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:40.565810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44230fc-806a-42b1-bbc9-f5727a197854 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:40.677547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44230fc-806a-42b1-bbc9-f5727a197854 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:40.714726 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7414241-8262-4bbe-a463-9354a4ff41e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:40.715386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7414241-8262-4bbe-a463-9354a4ff41e9 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:40.715679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7414241-8262-4bbe-a463-9354a4ff41e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:40.715952 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7414241-8262-4bbe-a463-9354a4ff41e9 None admin] PATH_INFO: `/v3/users/b562f81817e847deaa2210238e1bdd61` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:40.717138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7414241-8262-4bbe-a463-9354a4ff41e9 None admin] RBAC: Authorizing `identity:get_user(user_id=b562f81817e847deaa2210238e1bdd61)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:40.718571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7414241-8262-4bbe-a463-9354a4ff41e9 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:40.720234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 605/1190] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:56:40 2022] GET /identity/v3/users/b562f81817e847deaa2210238e1bdd61 => generated 311 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:40.726279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f7a1f0-9444-442a-af68-27555a2c2a1d None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:40.781568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:40.832719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f7a1f0-9444-442a-af68-27555a2c2a1d None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:40.869487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e8f0d6-b3eb-4025-b64b-4e2994b0059e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:40.870207 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e8f0d6-b3eb-4025-b64b-4e2994b0059e None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:40.870486 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e8f0d6-b3eb-4025-b64b-4e2994b0059e None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:40.870757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e8f0d6-b3eb-4025-b64b-4e2994b0059e None admin] PATH_INFO: `/v3/projects/7587c8f5812e4d36a314a8526b242f41` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:40.872000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e8f0d6-b3eb-4025-b64b-4e2994b0059e None admin] RBAC: Authorizing `identity:get_project(project_id=7587c8f5812e4d36a314a8526b242f41)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:40.873456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e8f0d6-b3eb-4025-b64b-4e2994b0059e None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:40.875186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 586/1191] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:56:40 2022] GET /identity/v3/projects/7587c8f5812e4d36a314a8526b242f41 => generated 361 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:56:40.881499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7414241-8262-4bbe-a463-9354a4ff41e9 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:40.926779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:40.998139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7414241-8262-4bbe-a463-9354a4ff41e9 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:41.039061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95ca3f5-cac0-4d72-98a1-6d10e9364f1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:41.039757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95ca3f5-cac0-4d72-98a1-6d10e9364f1e None admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:41.040036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95ca3f5-cac0-4d72-98a1-6d10e9364f1e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:41.040306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95ca3f5-cac0-4d72-98a1-6d10e9364f1e None admin] PATH_INFO: `/v3/projects/7587c8f5812e4d36a314a8526b242f41/users/b562f81817e847deaa2210238e1bdd61/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:41.042352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95ca3f5-cac0-4d72-98a1-6d10e9364f1e None admin] RBAC: Authorizing `identity:create_grant(project_id=7587c8f5812e4d36a314a8526b242f41, user_id=b562f81817e847deaa2210238e1bdd61, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:41.043813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95ca3f5-cac0-4d72-98a1-6d10e9364f1e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:41.058177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 606/1192] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:56:40 2022] PUT /identity/v3/projects/7587c8f5812e4d36a314a8526b242f41/users/b562f81817e847deaa2210238e1bdd61/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:56:43.129662 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341b8895-606c-44f6-b48c-e685220addae None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:43.130221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341b8895-606c-44f6-b48c-e685220addae None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:43.130221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341b8895-606c-44f6-b48c-e685220addae None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:43.131680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 587/1193] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:56:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:56:43.137739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a05b99-8f05-45c2-93a3-91b58af60bf0 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:43.138223 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a05b99-8f05-45c2-93a3-91b58af60bf0 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:43.138352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a05b99-8f05-45c2-93a3-91b58af60bf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:43.157202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b4a05b99-8f05-45c2-93a3-91b58af60bf0 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:56:43.158437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b4a05b99-8f05-45c2-93a3-91b58af60bf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:56:43.226908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 607/1194] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:56:43 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:56:43.250349 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341b8895-606c-44f6-b48c-e685220addae None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:43.356860 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341b8895-606c-44f6-b48c-e685220addae None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:43.379509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:43.395590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1907daab-d76b-4ba4-8d69-27c8d4cc5d34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:43.396153 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1907daab-d76b-4ba4-8d69-27c8d4cc5d34 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:43.396426 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1907daab-d76b-4ba4-8d69-27c8d4cc5d34 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:43.396695 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1907daab-d76b-4ba4-8d69-27c8d4cc5d34 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:43.412044 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1907daab-d76b-4ba4-8d69-27c8d4cc5d34 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-1907daab-d76b-4ba4-8d69-27c8d4cc5d34 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:56:43.417134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:56:43.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:56:43.420537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 588/1195] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:56:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:56:43.426249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a05b99-8f05-45c2-93a3-91b58af60bf0 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:43.461305 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:43.537091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a05b99-8f05-45c2-93a3-91b58af60bf0 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:43.572185 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659919d-1762-40d1-92e7-2db4105fe705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:43.572794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659919d-1762-40d1-92e7-2db4105fe705 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:43.573676 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659919d-1762-40d1-92e7-2db4105fe705 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:43.573949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659919d-1762-40d1-92e7-2db4105fe705 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:43.574661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9659919d-1762-40d1-92e7-2db4105fe705 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:43.574960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9659919d-1762-40d1-92e7-2db4105fe705 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:43.576354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9659919d-1762-40d1-92e7-2db4105fe705 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:43.584869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 608/1196] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:56:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:43.591740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1907daab-d76b-4ba4-8d69-27c8d4cc5d34 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:43.732137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1907daab-d76b-4ba4-8d69-27c8d4cc5d34 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:43.771248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e581b858-0879-4910-b915-5a307529d5de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:43.771943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e581b858-0879-4910-b915-5a307529d5de None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:43.772222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e581b858-0879-4910-b915-5a307529d5de None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:43.772492 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e581b858-0879-4910-b915-5a307529d5de None admin] PATH_INFO: `/v3/users/b562f81817e847deaa2210238e1bdd61` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:43.773742 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e581b858-0879-4910-b915-5a307529d5de None admin] RBAC: Authorizing `identity:get_user(user_id=b562f81817e847deaa2210238e1bdd61)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:43.775218 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e581b858-0879-4910-b915-5a307529d5de None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:43.777031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 589/1197] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Jul 31 00:56:43 2022] GET /identity/v3/users/b562f81817e847deaa2210238e1bdd61 => generated 311 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:43.783592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659919d-1762-40d1-92e7-2db4105fe705 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:43.881951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:43.891157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659919d-1762-40d1-92e7-2db4105fe705 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:43.926058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c540fb99-57fb-4cf2-a952-0dc5f5213498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:43.926675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c540fb99-57fb-4cf2-a952-0dc5f5213498 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:43.926956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c540fb99-57fb-4cf2-a952-0dc5f5213498 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:43.927232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c540fb99-57fb-4cf2-a952-0dc5f5213498 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:44.003910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c540fb99-57fb-4cf2-a952-0dc5f5213498 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:44.005300 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c540fb99-57fb-4cf2-a952-0dc5f5213498 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:44.101888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:44.113709 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 609/1198] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:56:43 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:56:44.120029 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e581b858-0879-4910-b915-5a307529d5de None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:44.172908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:44.227546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e581b858-0879-4910-b915-5a307529d5de None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:44.262934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a8e0d8-6013-4cba-b4f8-a17468a9df4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:44.263507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8e0d8-6013-4cba-b4f8-a17468a9df4c None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:44.263783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8e0d8-6013-4cba-b4f8-a17468a9df4c None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:44.264050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8e0d8-6013-4cba-b4f8-a17468a9df4c None admin] PATH_INFO: `/v3/projects/7587c8f5812e4d36a314a8526b242f41` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:44.265210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a8e0d8-6013-4cba-b4f8-a17468a9df4c None admin] RBAC: Authorizing `identity:get_project(project_id=7587c8f5812e4d36a314a8526b242f41)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:44.267033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a8e0d8-6013-4cba-b4f8-a17468a9df4c None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:44.268765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 590/1199] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Jul 31 00:56:44 2022] GET /identity/v3/projects/7587c8f5812e4d36a314a8526b242f41 => generated 361 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:56:44.275387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c540fb99-57fb-4cf2-a952-0dc5f5213498 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:56:44.379137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c540fb99-57fb-4cf2-a952-0dc5f5213498 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:56:44.416077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b7827e-a2d2-4299-a9d5-b45a58c973cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:56:44.416756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b7827e-a2d2-4299-a9d5-b45a58c973cf None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:56:44.417543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b7827e-a2d2-4299-a9d5-b45a58c973cf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:56:44.417807 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b7827e-a2d2-4299-a9d5-b45a58c973cf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:56:44.418597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b7827e-a2d2-4299-a9d5-b45a58c973cf None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=7587c8f5812e4d36a314a8526b242f41, user.id=b562f81817e847deaa2210238e1bdd61) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:56:44.418900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b7827e-a2d2-4299-a9d5-b45a58c973cf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:56:44.420378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b7827e-a2d2-4299-a9d5-b45a58c973cf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:56:44.427234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 610/1200] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:56:44 2022] GET /identity/v3/role_assignments?user.id=b562f81817e847deaa2210238e1bdd61&scope.project.id=7587c8f5812e4d36a314a8526b242f41&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:56:54.258675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=42 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:56:54.424207 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=72 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:41.256295 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938b3170-5961-4ae0-831f-35662eb55462 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:41.256957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938b3170-5961-4ae0-831f-35662eb55462 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:41.256957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938b3170-5961-4ae0-831f-35662eb55462 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:41.259035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 591/1201] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:57:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:57:41.264800 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b69fb4-5e5a-4129-8afc-23e3306580a4 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:41.265319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b69fb4-5e5a-4129-8afc-23e3306580a4 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:41.265319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b69fb4-5e5a-4129-8afc-23e3306580a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:41.285739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-46b69fb4-5e5a-4129-8afc-23e3306580a4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:57:41.287277 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-46b69fb4-5e5a-4129-8afc-23e3306580a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:57:41.367686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 611/1202] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:57:41 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 31 00:57:41.394927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938b3170-5961-4ae0-831f-35662eb55462 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:41.532428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938b3170-5961-4ae0-831f-35662eb55462 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:41.580799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382beb5d-e758-46ed-9449-265f98c2dbd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:41.581642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382beb5d-e758-46ed-9449-265f98c2dbd6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:41.581861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382beb5d-e758-46ed-9449-265f98c2dbd6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:41.582133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382beb5d-e758-46ed-9449-265f98c2dbd6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:41.594217 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382beb5d-e758-46ed-9449-265f98c2dbd6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:41.595727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382beb5d-e758-46ed-9449-265f98c2dbd6 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-382beb5d-e758-46ed-9449-265f98c2dbd6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:57:41.613709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:41.614966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:41.616541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:57:41.616541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:57:41.616541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:41.616541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:57:41.616541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 592/1203] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:57:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:41.621378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b69fb4-5e5a-4129-8afc-23e3306580a4 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:41.748443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b69fb4-5e5a-4129-8afc-23e3306580a4 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:41.791931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:41.801818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff37d291-fa55-46cb-9ace-dd2fe9e3d5c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:41.802520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff37d291-fa55-46cb-9ace-dd2fe9e3d5c1 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:41.802760 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff37d291-fa55-46cb-9ace-dd2fe9e3d5c1 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:41.803026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff37d291-fa55-46cb-9ace-dd2fe9e3d5c1 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:41.803782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff37d291-fa55-46cb-9ace-dd2fe9e3d5c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:41.804101 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff37d291-fa55-46cb-9ace-dd2fe9e3d5c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:41.805552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff37d291-fa55-46cb-9ace-dd2fe9e3d5c1 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:41.823396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 612/1204] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:57:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:41.825225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382beb5d-e758-46ed-9449-265f98c2dbd6 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:41.892949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:41.962645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382beb5d-e758-46ed-9449-265f98c2dbd6 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:42.007507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f501f00-eb12-4a13-8da9-e90123a900ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:42.008316 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f501f00-eb12-4a13-8da9-e90123a900ee None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:42.008595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f501f00-eb12-4a13-8da9-e90123a900ee None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:42.008863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f501f00-eb12-4a13-8da9-e90123a900ee None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:42.009719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f501f00-eb12-4a13-8da9-e90123a900ee None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:42.011341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f501f00-eb12-4a13-8da9-e90123a900ee None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:42.051963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 593/1205] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Jul 31 00:57:41 2022] POST /identity/v3/users => generated 251 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:57:44.278920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e9068e-26db-4d20-b644-f9077f142147 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:44.279536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e9068e-26db-4d20-b644-f9077f142147 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:44.279536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e9068e-26db-4d20-b644-f9077f142147 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:44.280995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 613/1206] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:57:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:57:44.286397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc01a9-dbcc-45d1-a82a-13e6d96d431e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:44.286901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc01a9-dbcc-45d1-a82a-13e6d96d431e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:44.287007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc01a9-dbcc-45d1-a82a-13e6d96d431e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:44.306560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d0bc01a9-dbcc-45d1-a82a-13e6d96d431e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:57:44.307791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d0bc01a9-dbcc-45d1-a82a-13e6d96d431e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:57:44.380383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 594/1207] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:57:44 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:57:44.406922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e9068e-26db-4d20-b644-f9077f142147 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:44.524156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e9068e-26db-4d20-b644-f9077f142147 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:44.567340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6658d6e3-5255-4178-9668-34f6f3434a43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:44.568016 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6658d6e3-5255-4178-9668-34f6f3434a43 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:44.568289 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6658d6e3-5255-4178-9668-34f6f3434a43 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:44.568554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6658d6e3-5255-4178-9668-34f6f3434a43 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:44.580461 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6658d6e3-5255-4178-9668-34f6f3434a43 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-6658d6e3-5255-4178-9668-34f6f3434a43 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:57:44.585403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:44.586542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:57:44.587597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 614/1208] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:57:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:44.595084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bc01a9-dbcc-45d1-a82a-13e6d96d431e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:44.646289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:44.712632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bc01a9-dbcc-45d1-a82a-13e6d96d431e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:44.752792 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa5d42a-314a-4d59-bf2d-2b0d1bf142f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:44.753437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa5d42a-314a-4d59-bf2d-2b0d1bf142f2 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:44.753750 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa5d42a-314a-4d59-bf2d-2b0d1bf142f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:44.754067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa5d42a-314a-4d59-bf2d-2b0d1bf142f2 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:44.754736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa5d42a-314a-4d59-bf2d-2b0d1bf142f2 None admin] RBAC: Adding query filter params (name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:44.755042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa5d42a-314a-4d59-bf2d-2b0d1bf142f2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:44.756552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa5d42a-314a-4d59-bf2d-2b0d1bf142f2 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:44.765908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 595/1209] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:57:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:57:44.771930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6658d6e3-5255-4178-9668-34f6f3434a43 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:44.830864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:44.891806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6658d6e3-5255-4178-9668-34f6f3434a43 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:44.936487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94fb554-2daa-4b51-9f97-1af6bd561fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:44.937088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94fb554-2daa-4b51-9f97-1af6bd561fc8 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:44.937363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94fb554-2daa-4b51-9f97-1af6bd561fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:44.937639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94fb554-2daa-4b51-9f97-1af6bd561fc8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:44.948888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94fb554-2daa-4b51-9f97-1af6bd561fc8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:44.950364 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94fb554-2daa-4b51-9f97-1af6bd561fc8 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-b94fb554-2daa-4b51-9f97-1af6bd561fc8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:57:44.966373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:44.967600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:44.968558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:57:44.968558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:57:44.968558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:44.968558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:57:44.968558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 615/1210] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:57:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:44.975280 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa5d42a-314a-4d59-bf2d-2b0d1bf142f2 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:45.106541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:45.110141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa5d42a-314a-4d59-bf2d-2b0d1bf142f2 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:45.155299 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14216f8e-4e0b-4aff-a0b2-1451dea577a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:45.156013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14216f8e-4e0b-4aff-a0b2-1451dea577a0 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:45.156297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14216f8e-4e0b-4aff-a0b2-1451dea577a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:45.156586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14216f8e-4e0b-4aff-a0b2-1451dea577a0 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:45.157294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14216f8e-4e0b-4aff-a0b2-1451dea577a0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:45.157664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14216f8e-4e0b-4aff-a0b2-1451dea577a0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:45.159037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14216f8e-4e0b-4aff-a0b2-1451dea577a0 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:45.180102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 596/1211] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:57:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:45.181761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94fb554-2daa-4b51-9f97-1af6bd561fc8 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:45.317834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94fb554-2daa-4b51-9f97-1af6bd561fc8 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:45.320630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:45.365594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb57423-d6fc-4737-859f-e4b5d24ce0fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:45.365594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb57423-d6fc-4737-859f-e4b5d24ce0fe None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:45.365871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb57423-d6fc-4737-859f-e4b5d24ce0fe None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:45.365871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb57423-d6fc-4737-859f-e4b5d24ce0fe None admin] PATH_INFO: `/v3/users/placement` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:45.375045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb57423-d6fc-4737-859f-e4b5d24ce0fe None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:45.376538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb57423-d6fc-4737-859f-e4b5d24ce0fe None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-bcb57423-d6fc-4737-859f-e4b5d24ce0fe None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:57:45.390618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:45.391740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:45.392703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:45.392703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:45.392703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:57:45.392703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:57:45.392703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:57:45.392703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:57:45.392703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:57:45.392703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:57:45.392703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 31 00:57:45.392703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:57:45.392703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 616/1212] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:57:45 2022] GET /identity/v3/users/placement => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:57:45.398283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14216f8e-4e0b-4aff-a0b2-1451dea577a0 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:45.567734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14216f8e-4e0b-4aff-a0b2-1451dea577a0 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:45.624318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8a8dbd-d305-412b-8de2-bec74cf37675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:45.624980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8a8dbd-d305-412b-8de2-bec74cf37675 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:45.625351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8a8dbd-d305-412b-8de2-bec74cf37675 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:45.625599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8a8dbd-d305-412b-8de2-bec74cf37675 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:45.626556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8a8dbd-d305-412b-8de2-bec74cf37675 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:45.626942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8a8dbd-d305-412b-8de2-bec74cf37675 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:45.628652 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8a8dbd-d305-412b-8de2-bec74cf37675 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:45.651439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 597/1213] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Jul 31 00:57:45 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:45.659464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb57423-d6fc-4737-859f-e4b5d24ce0fe None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:45.814302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb57423-d6fc-4737-859f-e4b5d24ce0fe None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:45.857104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee04b31-8299-45af-b40d-a0b514db1928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:45.858071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee04b31-8299-45af-b40d-a0b514db1928 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:45.858509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee04b31-8299-45af-b40d-a0b514db1928 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:45.858988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee04b31-8299-45af-b40d-a0b514db1928 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:45.897500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:45.952864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee04b31-8299-45af-b40d-a0b514db1928 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:45.954663 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee04b31-8299-45af-b40d-a0b514db1928 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:46.113928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 617/1214] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:57:45 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 31 00:57:46.121770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8a8dbd-d305-412b-8de2-bec74cf37675 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:46.155511 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:46.268415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8a8dbd-d305-412b-8de2-bec74cf37675 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:46.315172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04057486-f88a-40fe-bc6a-f70512c652f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:46.316266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04057486-f88a-40fe-bc6a-f70512c652f9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:46.316433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04057486-f88a-40fe-bc6a-f70512c652f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:46.316708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04057486-f88a-40fe-bc6a-f70512c652f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:46.331479 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04057486-f88a-40fe-bc6a-f70512c652f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:46.333710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04057486-f88a-40fe-bc6a-f70512c652f9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-04057486-f88a-40fe-bc6a-f70512c652f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:57:46.355164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:46.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:46.357300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:57:46.357300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:57:46.357300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:46.357300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:57:46.357300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 598/1215] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:57:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:46.363030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee04b31-8299-45af-b40d-a0b514db1928 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:46.432432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:46.489394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee04b31-8299-45af-b40d-a0b514db1928 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:46.561208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2749f15-8236-4a40-a358-a060bb0357cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:46.562311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2749f15-8236-4a40-a358-a060bb0357cf None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:46.562778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2749f15-8236-4a40-a358-a060bb0357cf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:46.563244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2749f15-8236-4a40-a358-a060bb0357cf None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:46.564573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2749f15-8236-4a40-a358-a060bb0357cf None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:46.565139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2749f15-8236-4a40-a358-a060bb0357cf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:46.567757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2749f15-8236-4a40-a358-a060bb0357cf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:46.595396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 618/1216] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:57:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:46.599883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04057486-f88a-40fe-bc6a-f70512c652f9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:46.687483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:46.739675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04057486-f88a-40fe-bc6a-f70512c652f9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:46.795161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2939e9e9-580a-44b9-9740-6c70140580d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:46.796177 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2939e9e9-580a-44b9-9740-6c70140580d4 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:46.797641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2939e9e9-580a-44b9-9740-6c70140580d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:46.797641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2939e9e9-580a-44b9-9740-6c70140580d4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:46.805169 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2939e9e9-580a-44b9-9740-6c70140580d4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:46.806808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2939e9e9-580a-44b9-9740-6c70140580d4 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-2939e9e9-580a-44b9-9740-6c70140580d4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:57:46.819043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:46.820721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:46.822170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:57:46.822170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:57:46.822170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:57:46.822170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:57:46.822170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:57:46.822170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:57:46.822170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:57:46.822170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:57:46.822170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 599/1217] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:57:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:57:46.827025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2749f15-8236-4a40-a358-a060bb0357cf None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:46.983288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2749f15-8236-4a40-a358-a060bb0357cf None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:47.013203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:47.029497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd82bd1a-9b00-4bf7-81fd-44eea2292a70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:47.030186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd82bd1a-9b00-4bf7-81fd-44eea2292a70 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:47.030495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd82bd1a-9b00-4bf7-81fd-44eea2292a70 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:47.030816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd82bd1a-9b00-4bf7-81fd-44eea2292a70 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:47.031692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd82bd1a-9b00-4bf7-81fd-44eea2292a70 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:47.031973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd82bd1a-9b00-4bf7-81fd-44eea2292a70 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:47.033565 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd82bd1a-9b00-4bf7-81fd-44eea2292a70 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:47.054190 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 619/1218] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:57:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:57:47.056398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2939e9e9-580a-44b9-9740-6c70140580d4 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:47.176881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2939e9e9-580a-44b9-9740-6c70140580d4 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:47.208787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:47.215301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1614b39-98b4-42d7-959f-67383efcac55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:47.216088 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1614b39-98b4-42d7-959f-67383efcac55 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:47.216544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1614b39-98b4-42d7-959f-67383efcac55 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:47.216931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1614b39-98b4-42d7-959f-67383efcac55 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:47.217827 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1614b39-98b4-42d7-959f-67383efcac55 None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=1d562ccd50e4492281a9f435f809d49b) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:47.218519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1614b39-98b4-42d7-959f-67383efcac55 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:47.219847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1614b39-98b4-42d7-959f-67383efcac55 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:47.235525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 600/1219] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:57:47 2022] GET /identity/v3/role_assignments?user.id=1d562ccd50e4492281a9f435f809d49b&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:49.297149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff56dd18-6425-439d-a8f7-8af53a24e222 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:49.297689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff56dd18-6425-439d-a8f7-8af53a24e222 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:49.297689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff56dd18-6425-439d-a8f7-8af53a24e222 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:49.299027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 620/1220] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:57:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:57:49.304455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2d454d-cf1e-4802-be2d-0ff5e4844374 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:49.304930 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2d454d-cf1e-4802-be2d-0ff5e4844374 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:49.305020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2d454d-cf1e-4802-be2d-0ff5e4844374 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:49.324196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6c2d454d-cf1e-4802-be2d-0ff5e4844374 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:57:49.325409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6c2d454d-cf1e-4802-be2d-0ff5e4844374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:57:49.420495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 601/1221] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:57:49 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:57:49.450402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff56dd18-6425-439d-a8f7-8af53a24e222 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:49.578686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff56dd18-6425-439d-a8f7-8af53a24e222 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:49.614108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688de850-2515-4df7-b2d7-7592312f2f4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:49.614741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688de850-2515-4df7-b2d7-7592312f2f4b None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:49.615016 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688de850-2515-4df7-b2d7-7592312f2f4b None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:49.615290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688de850-2515-4df7-b2d7-7592312f2f4b None admin] PATH_INFO: `/v3/roles/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:49.626322 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688de850-2515-4df7-b2d7-7592312f2f4b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-688de850-2515-4df7-b2d7-7592312f2f4b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:57:49.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:49.632949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:57:49.633833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 621/1222] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:57:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:57:49.639632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2d454d-cf1e-4802-be2d-0ff5e4844374 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:49.753398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2d454d-cf1e-4802-be2d-0ff5e4844374 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:49.780353 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:49.792639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbb7344-a30f-4fc9-a41e-d14a47876ff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:49.793259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbb7344-a30f-4fc9-a41e-d14a47876ff9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:49.793536 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbb7344-a30f-4fc9-a41e-d14a47876ff9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:49.793805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbb7344-a30f-4fc9-a41e-d14a47876ff9 None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:49.794550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbb7344-a30f-4fc9-a41e-d14a47876ff9 None admin] RBAC: Adding query filter params (name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:49.794858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbb7344-a30f-4fc9-a41e-d14a47876ff9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:49.796348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbb7344-a30f-4fc9-a41e-d14a47876ff9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:49.806249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 602/1223] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:57:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:49.811885 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688de850-2515-4df7-b2d7-7592312f2f4b None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:49.858368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:49.923399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688de850-2515-4df7-b2d7-7592312f2f4b None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:49.961922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672effe6-69d4-46b9-8420-689304b04184 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:49.962610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672effe6-69d4-46b9-8420-689304b04184 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:49.962937 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672effe6-69d4-46b9-8420-689304b04184 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:49.963263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672effe6-69d4-46b9-8420-689304b04184 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:49.974721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672effe6-69d4-46b9-8420-689304b04184 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:49.976429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672effe6-69d4-46b9-8420-689304b04184 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-672effe6-69d4-46b9-8420-689304b04184 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:57:49.994031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:49.995257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:49.996971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:57:49.996971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:57:49.996971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:49.996971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:57:49.996971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 622/1224] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:57:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:50.001693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbb7344-a30f-4fc9-a41e-d14a47876ff9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:50.119761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbb7344-a30f-4fc9-a41e-d14a47876ff9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:50.157158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d4918d-d701-4de7-92f8-16e096d28a64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:50.157867 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d4918d-d701-4de7-92f8-16e096d28a64 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:50.158244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d4918d-d701-4de7-92f8-16e096d28a64 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:50.158630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d4918d-d701-4de7-92f8-16e096d28a64 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:50.159487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d4918d-d701-4de7-92f8-16e096d28a64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:50.159929 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d4918d-d701-4de7-92f8-16e096d28a64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:50.161423 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d4918d-d701-4de7-92f8-16e096d28a64 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:50.174747 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 603/1225] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:57:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:50.181001 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672effe6-69d4-46b9-8420-689304b04184 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:50.280996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:50.289881 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672effe6-69d4-46b9-8420-689304b04184 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:50.330079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09333274-582b-469b-bbc6-f34db24edb93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:50.330765 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09333274-582b-469b-bbc6-f34db24edb93 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:50.330913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09333274-582b-469b-bbc6-f34db24edb93 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:50.331270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09333274-582b-469b-bbc6-f34db24edb93 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:50.340502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09333274-582b-469b-bbc6-f34db24edb93 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:50.342035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09333274-582b-469b-bbc6-f34db24edb93 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-09333274-582b-469b-bbc6-f34db24edb93 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:57:50.354035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:50.355130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:50.356459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:50.356459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:50.356459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:57:50.356459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:57:50.356459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:57:50.356459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:57:50.356459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:57:50.356459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:57:50.356459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 31 00:57:50.356459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:57:50.356459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 623/1226] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:57:50 2022] GET /identity/v3/users/placement => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:50.361322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d4918d-d701-4de7-92f8-16e096d28a64 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:50.419345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:50.474592 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d4918d-d701-4de7-92f8-16e096d28a64 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:50.509776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065b1d6f-205b-4c98-bca9-22579f0aa8fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:50.510394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065b1d6f-205b-4c98-bca9-22579f0aa8fc None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:50.510664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065b1d6f-205b-4c98-bca9-22579f0aa8fc None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:50.510971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065b1d6f-205b-4c98-bca9-22579f0aa8fc None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:50.511745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065b1d6f-205b-4c98-bca9-22579f0aa8fc None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:50.512042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065b1d6f-205b-4c98-bca9-22579f0aa8fc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:50.513465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065b1d6f-205b-4c98-bca9-22579f0aa8fc None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:50.533092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 604/1227] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Jul 31 00:57:50 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:50.539635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09333274-582b-469b-bbc6-f34db24edb93 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:50.644193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09333274-582b-469b-bbc6-f34db24edb93 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:50.683488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21fecc0-ebc7-4ae7-b21d-1389a908296d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:50.684063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21fecc0-ebc7-4ae7-b21d-1389a908296d None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:50.684344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21fecc0-ebc7-4ae7-b21d-1389a908296d None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:50.684616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21fecc0-ebc7-4ae7-b21d-1389a908296d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:50.697063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21fecc0-ebc7-4ae7-b21d-1389a908296d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:50.698597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21fecc0-ebc7-4ae7-b21d-1389a908296d None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-a21fecc0-ebc7-4ae7-b21d-1389a908296d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:57:50.714039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:50.715220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:50.716286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:57:50.716286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:57:50.716286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:50.716286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:57:50.716286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 624/1228] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:57:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:50.724471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065b1d6f-205b-4c98-bca9-22579f0aa8fc None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:50.834289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065b1d6f-205b-4c98-bca9-22579f0aa8fc None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:50.837368 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:50.871718 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05a8953-df43-4a72-9ae1-6cd3170e2c83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:50.872343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05a8953-df43-4a72-9ae1-6cd3170e2c83 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:50.872624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05a8953-df43-4a72-9ae1-6cd3170e2c83 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:50.872899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05a8953-df43-4a72-9ae1-6cd3170e2c83 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:50.873593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05a8953-df43-4a72-9ae1-6cd3170e2c83 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:50.873922 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05a8953-df43-4a72-9ae1-6cd3170e2c83 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:50.875301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05a8953-df43-4a72-9ae1-6cd3170e2c83 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:50.893202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 605/1229] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:57:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:50.894321 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21fecc0-ebc7-4ae7-b21d-1389a908296d None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:50.906812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:51.005792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21fecc0-ebc7-4ae7-b21d-1389a908296d None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:51.044313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dd901b-73da-41ab-8f2e-d0bb5310d195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:51.045015 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dd901b-73da-41ab-8f2e-d0bb5310d195 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:51.045294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dd901b-73da-41ab-8f2e-d0bb5310d195 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:51.045788 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dd901b-73da-41ab-8f2e-d0bb5310d195 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:51.054817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dd901b-73da-41ab-8f2e-d0bb5310d195 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:51.056330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dd901b-73da-41ab-8f2e-d0bb5310d195 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-59dd901b-73da-41ab-8f2e-d0bb5310d195 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:57:51.069563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:51.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:51.073787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:57:51.073787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:57:51.073787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:57:51.073787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:57:51.073787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:57:51.073787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:57:51.073787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:57:51.073787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:57:51.073787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 625/1230] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:57:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:51.076624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05a8953-df43-4a72-9ae1-6cd3170e2c83 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:51.199761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05a8953-df43-4a72-9ae1-6cd3170e2c83 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:51.236337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61e002a-014e-4463-bae2-ecbd29e4feb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:51.237026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61e002a-014e-4463-bae2-ecbd29e4feb2 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:51.237271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61e002a-014e-4463-bae2-ecbd29e4feb2 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:51.237562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61e002a-014e-4463-bae2-ecbd29e4feb2 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:51.238275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61e002a-014e-4463-bae2-ecbd29e4feb2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:51.238583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61e002a-014e-4463-bae2-ecbd29e4feb2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:51.240071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61e002a-014e-4463-bae2-ecbd29e4feb2 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:51.254917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 606/1231] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:57:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:51.261368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dd901b-73da-41ab-8f2e-d0bb5310d195 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:51.340786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:51.384745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dd901b-73da-41ab-8f2e-d0bb5310d195 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:51.428213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e29a2d-8572-4220-a764-33834d1335bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:51.428916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e29a2d-8572-4220-a764-33834d1335bc None admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:51.429195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e29a2d-8572-4220-a764-33834d1335bc None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:51.429478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e29a2d-8572-4220-a764-33834d1335bc None admin] PATH_INFO: `/v3/projects/8b41573c9f3146d19a306089d686015c/users/1d562ccd50e4492281a9f435f809d49b/roles/26925162cb82467f981eae19826e00c9` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:51.448163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e29a2d-8572-4220-a764-33834d1335bc None admin] RBAC: Authorizing `identity:create_grant(project_id=8b41573c9f3146d19a306089d686015c, user_id=1d562ccd50e4492281a9f435f809d49b, role_id=26925162cb82467f981eae19826e00c9)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:51.449736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e29a2d-8572-4220-a764-33834d1335bc None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:51.464794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 626/1232] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:57:51 2022] PUT /identity/v3/projects/8b41573c9f3146d19a306089d686015c/users/1d562ccd50e4492281a9f435f809d49b/roles/26925162cb82467f981eae19826e00c9 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:57:53.611744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1a2030-d529-4cf3-afd4-9a1c0312ff2e None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:53.612290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1a2030-d529-4cf3-afd4-9a1c0312ff2e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:53.612290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1a2030-d529-4cf3-afd4-9a1c0312ff2e None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:53.613818 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 607/1233] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:57:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:57:53.620892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0438df-dd5f-4ac6-b4b8-a143078f4e0b None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:53.621431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0438df-dd5f-4ac6-b4b8-a143078f4e0b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:53.621431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0438df-dd5f-4ac6-b4b8-a143078f4e0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:53.642244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-4d0438df-dd5f-4ac6-b4b8-a143078f4e0b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:57:53.643520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4d0438df-dd5f-4ac6-b4b8-a143078f4e0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:57:53.715227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 627/1234] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:57:53 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:57:53.738261 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1a2030-d529-4cf3-afd4-9a1c0312ff2e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:53.770854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:53.879446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1a2030-d529-4cf3-afd4-9a1c0312ff2e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:53.921575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a3fde5-db0d-4be9-9ce1-5b142bf7d2c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:53.922437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a3fde5-db0d-4be9-9ce1-5b142bf7d2c0 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:53.922740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a3fde5-db0d-4be9-9ce1-5b142bf7d2c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:53.922994 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a3fde5-db0d-4be9-9ce1-5b142bf7d2c0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:53.934406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a3fde5-db0d-4be9-9ce1-5b142bf7d2c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-d8a3fde5-db0d-4be9-9ce1-5b142bf7d2c0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:57:53.939580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:53.940774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:53.941861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:53.941861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:53.941861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:53.941861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:53.941861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:57:53.941861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:57:53.941861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:57:53.941861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:57:53.941861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:57:53.941861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:57:53.941861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 31 00:57:53.941861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:57:53.947984 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 608/1235] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Jul 31 00:57:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:53.948123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0438df-dd5f-4ac6-b4b8-a143078f4e0b None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:54.001640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:54.061811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0438df-dd5f-4ac6-b4b8-a143078f4e0b None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:54.102491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff381f6-4f13-4bb4-afe7-07900ee14066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:54.103262 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff381f6-4f13-4bb4-afe7-07900ee14066 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:54.103509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff381f6-4f13-4bb4-afe7-07900ee14066 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:54.103763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff381f6-4f13-4bb4-afe7-07900ee14066 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:54.104494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff381f6-4f13-4bb4-afe7-07900ee14066 None admin] RBAC: Adding query filter params (name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:54.104882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff381f6-4f13-4bb4-afe7-07900ee14066 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:54.106199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff381f6-4f13-4bb4-afe7-07900ee14066 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:54.115150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 628/1236] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Jul 31 00:57:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:57:54.122134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a3fde5-db0d-4be9-9ce1-5b142bf7d2c0 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:54.227439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:54.245742 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a3fde5-db0d-4be9-9ce1-5b142bf7d2c0 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:54.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a18a890-a33c-412e-b4b1-fd489dc0d288 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:54.284268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a18a890-a33c-412e-b4b1-fd489dc0d288 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:54.284655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a18a890-a33c-412e-b4b1-fd489dc0d288 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:54.284939 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a18a890-a33c-412e-b4b1-fd489dc0d288 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:54.297416 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a18a890-a33c-412e-b4b1-fd489dc0d288 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:54.298916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a18a890-a33c-412e-b4b1-fd489dc0d288 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-4a18a890-a33c-412e-b4b1-fd489dc0d288 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:57:54.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:54.316767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:54.317658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:57:54.317658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:57:54.317658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:54.317658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:57:54.317658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 609/1237] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:57:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:54.323083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff381f6-4f13-4bb4-afe7-07900ee14066 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:54.435616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff381f6-4f13-4bb4-afe7-07900ee14066 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:54.445824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:54.477553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8bdfcf-6c21-442f-8a3c-aacf6d1b6b1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:54.478345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8bdfcf-6c21-442f-8a3c-aacf6d1b6b1e None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:54.478849 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8bdfcf-6c21-442f-8a3c-aacf6d1b6b1e None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:54.479293 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8bdfcf-6c21-442f-8a3c-aacf6d1b6b1e None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:54.480222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8bdfcf-6c21-442f-8a3c-aacf6d1b6b1e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:54.480692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8bdfcf-6c21-442f-8a3c-aacf6d1b6b1e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:54.482131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8bdfcf-6c21-442f-8a3c-aacf6d1b6b1e None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:54.495455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 629/1238] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:57:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:54.501360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a18a890-a33c-412e-b4b1-fd489dc0d288 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:54.633654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a18a890-a33c-412e-b4b1-fd489dc0d288 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:54.671945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a88f451-c77e-400c-8d1d-c4059625fcbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:54.672585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a88f451-c77e-400c-8d1d-c4059625fcbf None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:54.672864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a88f451-c77e-400c-8d1d-c4059625fcbf None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:54.673127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a88f451-c77e-400c-8d1d-c4059625fcbf None admin] PATH_INFO: `/v3/users/placement` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:54.676087 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:54.681506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a88f451-c77e-400c-8d1d-c4059625fcbf None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:54.682957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a88f451-c77e-400c-8d1d-c4059625fcbf None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-6a88f451-c77e-400c-8d1d-c4059625fcbf None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:57:54.695931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:54.697150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:54.698226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:54.698226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:54.698226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:57:54.698226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:57:54.698226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:57:54.698226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:57:54.698226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:57:54.698226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:57:54.698226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 31 00:57:54.698226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:57:54.698226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 610/1239] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:57:54 2022] GET /identity/v3/users/placement => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:54.702935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8bdfcf-6c21-442f-8a3c-aacf6d1b6b1e None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:54.814788 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8bdfcf-6c21-442f-8a3c-aacf6d1b6b1e None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:54.863345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01266d9f-fcda-4d16-aa22-d41cf35e6807 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:54.864098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01266d9f-fcda-4d16-aa22-d41cf35e6807 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:54.864356 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01266d9f-fcda-4d16-aa22-d41cf35e6807 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:54.864726 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01266d9f-fcda-4d16-aa22-d41cf35e6807 None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:54.865757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01266d9f-fcda-4d16-aa22-d41cf35e6807 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:54.866108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01266d9f-fcda-4d16-aa22-d41cf35e6807 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:54.867898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01266d9f-fcda-4d16-aa22-d41cf35e6807 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:54.895625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 630/1240] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Jul 31 00:57:54 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:54.897984 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a88f451-c77e-400c-8d1d-c4059625fcbf None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:55.020048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a88f451-c77e-400c-8d1d-c4059625fcbf None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:55.058459 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cc3c30-22d3-4528-be33-86ef37d64c14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:55.059114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cc3c30-22d3-4528-be33-86ef37d64c14 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:55.059485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cc3c30-22d3-4528-be33-86ef37d64c14 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:55.059758 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cc3c30-22d3-4528-be33-86ef37d64c14 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:55.144790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:55.154800 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cc3c30-22d3-4528-be33-86ef37d64c14 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:55.156440 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cc3c30-22d3-4528-be33-86ef37d64c14 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:55.270574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 611/1241] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:57:54 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:57:55.276479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01266d9f-fcda-4d16-aa22-d41cf35e6807 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:55.308321 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:55.389543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01266d9f-fcda-4d16-aa22-d41cf35e6807 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:55.427902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be92ba2c-24c9-48bd-9e6f-e2123238772c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:55.429095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be92ba2c-24c9-48bd-9e6f-e2123238772c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:55.429374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be92ba2c-24c9-48bd-9e6f-e2123238772c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:55.429642 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be92ba2c-24c9-48bd-9e6f-e2123238772c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:55.440621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be92ba2c-24c9-48bd-9e6f-e2123238772c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:55.442103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be92ba2c-24c9-48bd-9e6f-e2123238772c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-be92ba2c-24c9-48bd-9e6f-e2123238772c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:57:55.458018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:55.459311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:55.460445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:57:55.460445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:57:55.460445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:55.460445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:57:55.460445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 631/1242] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:57:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:57:55.465852 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cc3c30-22d3-4528-be33-86ef37d64c14 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:55.587716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:55.594543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cc3c30-22d3-4528-be33-86ef37d64c14 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:55.631022 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c6c2ef-bf9c-4106-a1e0-d6190f10aed9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:55.631713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c6c2ef-bf9c-4106-a1e0-d6190f10aed9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:55.632000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c6c2ef-bf9c-4106-a1e0-d6190f10aed9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:55.632272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c6c2ef-bf9c-4106-a1e0-d6190f10aed9 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:55.632964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c6c2ef-bf9c-4106-a1e0-d6190f10aed9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:55.633265 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c6c2ef-bf9c-4106-a1e0-d6190f10aed9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:55.634661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c6c2ef-bf9c-4106-a1e0-d6190f10aed9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:55.646266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 612/1243] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:57:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:55.652204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be92ba2c-24c9-48bd-9e6f-e2123238772c None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:55.738791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:55.763657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be92ba2c-24c9-48bd-9e6f-e2123238772c None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:55.805590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e2c915-361f-4e27-a6c1-56081d94d2f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:55.806423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e2c915-361f-4e27-a6c1-56081d94d2f5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:55.806898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e2c915-361f-4e27-a6c1-56081d94d2f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:55.807347 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e2c915-361f-4e27-a6c1-56081d94d2f5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:55.815804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e2c915-361f-4e27-a6c1-56081d94d2f5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:55.817360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e2c915-361f-4e27-a6c1-56081d94d2f5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-43e2c915-361f-4e27-a6c1-56081d94d2f5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:57:55.829665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:55.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:55.831927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:57:55.831927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:57:55.831927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:57:55.831927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:57:55.831927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:57:55.831927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:57:55.831927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:57:55.831927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:57:55.831927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 632/1244] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:57:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:55.837573 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c6c2ef-bf9c-4106-a1e0-d6190f10aed9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:55.953543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c6c2ef-bf9c-4106-a1e0-d6190f10aed9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:55.998622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fe5491-9b89-48bc-91f7-54b39ff93a11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:55.999209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe5491-9b89-48bc-91f7-54b39ff93a11 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:55.999514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe5491-9b89-48bc-91f7-54b39ff93a11 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:55.999778 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fe5491-9b89-48bc-91f7-54b39ff93a11 None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:56.000519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fe5491-9b89-48bc-91f7-54b39ff93a11 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:56.000824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fe5491-9b89-48bc-91f7-54b39ff93a11 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:56.002267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fe5491-9b89-48bc-91f7-54b39ff93a11 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:56.016414 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 613/1245] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:57:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:56.023528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e2c915-361f-4e27-a6c1-56081d94d2f5 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:56.137459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e2c915-361f-4e27-a6c1-56081d94d2f5 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:56.179292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60996aff-ef7a-4ab7-b8de-4c45bf7cfbdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:56.180098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60996aff-ef7a-4ab7-b8de-4c45bf7cfbdf None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:56.180320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60996aff-ef7a-4ab7-b8de-4c45bf7cfbdf None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:56.180588 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60996aff-ef7a-4ab7-b8de-4c45bf7cfbdf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:56.181562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60996aff-ef7a-4ab7-b8de-4c45bf7cfbdf None admin] RBAC: Adding query filter params (role.id=26925162cb82467f981eae19826e00c9, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=1d562ccd50e4492281a9f435f809d49b) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:56.181920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60996aff-ef7a-4ab7-b8de-4c45bf7cfbdf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:56.184041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60996aff-ef7a-4ab7-b8de-4c45bf7cfbdf None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:56.190031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 633/1246] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:57:56 2022] GET /identity/v3/role_assignments?user.id=1d562ccd50e4492281a9f435f809d49b&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=26925162cb82467f981eae19826e00c9 => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:58.473779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41136106-ae3e-4c37-8e57-c21ff09ee6aa None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:58.474360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41136106-ae3e-4c37-8e57-c21ff09ee6aa None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:58.474360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41136106-ae3e-4c37-8e57-c21ff09ee6aa None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:58.476315 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 614/1247] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:57:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 00:57:58.482292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc987c64-fde0-4e41-a52b-901d6eae2ded None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:58.482753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc987c64-fde0-4e41-a52b-901d6eae2ded None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:58.482854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc987c64-fde0-4e41-a52b-901d6eae2ded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:58.490712 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:58.504643 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-fc987c64-fde0-4e41-a52b-901d6eae2ded None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:57:58.506013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-fc987c64-fde0-4e41-a52b-901d6eae2ded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:57:58.585262 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 634/1248] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:57:58 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:57:58.609203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41136106-ae3e-4c37-8e57-c21ff09ee6aa None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:58.630279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:58.728474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41136106-ae3e-4c37-8e57-c21ff09ee6aa None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:58.767864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92c750f-0b74-4ec1-a290-3158d9d8caab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:58.767864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92c750f-0b74-4ec1-a290-3158d9d8caab None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:58.767864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92c750f-0b74-4ec1-a290-3158d9d8caab None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:58.768359 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92c750f-0b74-4ec1-a290-3158d9d8caab None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:58.781209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92c750f-0b74-4ec1-a290-3158d9d8caab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-a92c750f-0b74-4ec1-a290-3158d9d8caab None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:57:58.786745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:58.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:57:58.793174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 615/1249] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:57:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:58.794751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc987c64-fde0-4e41-a52b-901d6eae2ded None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:58.902159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc987c64-fde0-4e41-a52b-901d6eae2ded None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:58.933843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:58.939747 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eed6f07-67d6-46b5-8016-17f55422fc94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:58.940392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eed6f07-67d6-46b5-8016-17f55422fc94 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:58.940669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eed6f07-67d6-46b5-8016-17f55422fc94 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:58.940956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eed6f07-67d6-46b5-8016-17f55422fc94 None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:58.941740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eed6f07-67d6-46b5-8016-17f55422fc94 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:58.942050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eed6f07-67d6-46b5-8016-17f55422fc94 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:58.943599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eed6f07-67d6-46b5-8016-17f55422fc94 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:58.952442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 635/1250] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:57:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:58.959570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92c750f-0b74-4ec1-a290-3158d9d8caab None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:59.054668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:59.083224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92c750f-0b74-4ec1-a290-3158d9d8caab None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:59.126782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805040d4-e88f-4bb1-8f02-d7e56535a3c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:59.127798 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805040d4-e88f-4bb1-8f02-d7e56535a3c6 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:59.127908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805040d4-e88f-4bb1-8f02-d7e56535a3c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:59.128199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805040d4-e88f-4bb1-8f02-d7e56535a3c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:59.141050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805040d4-e88f-4bb1-8f02-d7e56535a3c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:59.142800 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805040d4-e88f-4bb1-8f02-d7e56535a3c6 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-805040d4-e88f-4bb1-8f02-d7e56535a3c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:57:59.160549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:59.161567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:59.162503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:57:59.162503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:57:59.162503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:57:59.162503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:57:59.162503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 616/1251] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:57:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:59.168274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eed6f07-67d6-46b5-8016-17f55422fc94 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:59.283099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eed6f07-67d6-46b5-8016-17f55422fc94 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:59.321089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503e01e1-450c-435f-95ea-c161445a9047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:59.321599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503e01e1-450c-435f-95ea-c161445a9047 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:59.321874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503e01e1-450c-435f-95ea-c161445a9047 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:59.322143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503e01e1-450c-435f-95ea-c161445a9047 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:59.322831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503e01e1-450c-435f-95ea-c161445a9047 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:59.323130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503e01e1-450c-435f-95ea-c161445a9047 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:59.324601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503e01e1-450c-435f-95ea-c161445a9047 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:59.338489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 636/1252] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:57:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:59.344565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805040d4-e88f-4bb1-8f02-d7e56535a3c6 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:59.456692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805040d4-e88f-4bb1-8f02-d7e56535a3c6 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:59.486973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:59.497799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd9857-821c-4f8a-8d38-6733d3f2c231 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:59.498476 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd9857-821c-4f8a-8d38-6733d3f2c231 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:59.498785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd9857-821c-4f8a-8d38-6733d3f2c231 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:59.499144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd9857-821c-4f8a-8d38-6733d3f2c231 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:59.508881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cd9857-821c-4f8a-8d38-6733d3f2c231 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:59.510535 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cd9857-821c-4f8a-8d38-6733d3f2c231 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-40cd9857-821c-4f8a-8d38-6733d3f2c231 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:57:59.525255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:57:59.526424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:57:59.527558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:57:59.527558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:57:59.527558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:57:59.527558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:57:59.527558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:57:59.527558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:57:59.527558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:57:59.527558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:57:59.527558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 31 00:57:59.527558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:57:59.527558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 617/1253] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:57:59 2022] GET /identity/v3/users/placement => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:57:59.532749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503e01e1-450c-435f-95ea-c161445a9047 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:59.597665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:59.648252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503e01e1-450c-435f-95ea-c161445a9047 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:59.686124 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b232bc73-6c09-40cf-9a57-afe2091e6feb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:59.686846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b232bc73-6c09-40cf-9a57-afe2091e6feb None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:59.687020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b232bc73-6c09-40cf-9a57-afe2091e6feb None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:59.687377 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b232bc73-6c09-40cf-9a57-afe2091e6feb None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:59.688206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b232bc73-6c09-40cf-9a57-afe2091e6feb None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:57:59.688513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b232bc73-6c09-40cf-9a57-afe2091e6feb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:59.690044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b232bc73-6c09-40cf-9a57-afe2091e6feb None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:57:59.710613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 637/1254] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Jul 31 00:57:59 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:57:59.716819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd9857-821c-4f8a-8d38-6733d3f2c231 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:57:59.838848 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd9857-821c-4f8a-8d38-6733d3f2c231 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:57:59.879456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a3e597-7d36-4960-a242-c8978d234950 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:57:59.880001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a3e597-7d36-4960-a242-c8978d234950 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:57:59.880289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a3e597-7d36-4960-a242-c8978d234950 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:57:59.880553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a3e597-7d36-4960-a242-c8978d234950 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:57:59.941294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:57:59.954593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a3e597-7d36-4960-a242-c8978d234950 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:57:59.956098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a3e597-7d36-4960-a242-c8978d234950 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:00.087160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 618/1255] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:57:59 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:58:00.094817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b232bc73-6c09-40cf-9a57-afe2091e6feb None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:00.245073 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b232bc73-6c09-40cf-9a57-afe2091e6feb None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:00.263263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:00.291497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73665714-5d89-467e-adfb-796f513b7798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:00.292303 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73665714-5d89-467e-adfb-796f513b7798 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:00.292664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73665714-5d89-467e-adfb-796f513b7798 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:00.292664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73665714-5d89-467e-adfb-796f513b7798 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:00.304028 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73665714-5d89-467e-adfb-796f513b7798 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:00.305758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73665714-5d89-467e-adfb-796f513b7798 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-73665714-5d89-467e-adfb-796f513b7798 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:58:00.325108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:00.326202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:00.327209 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:58:00.327209 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:58:00.327209 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:58:00.327209 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:00.333158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 638/1256] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:58:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:00.335318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a3e597-7d36-4960-a242-c8978d234950 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:00.454891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:00.470622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a3e597-7d36-4960-a242-c8978d234950 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:00.508143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac71591-d163-4dd8-8a2f-bbf13ab9216f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:00.508782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac71591-d163-4dd8-8a2f-bbf13ab9216f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:00.509080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac71591-d163-4dd8-8a2f-bbf13ab9216f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:00.509354 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac71591-d163-4dd8-8a2f-bbf13ab9216f None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:00.510045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac71591-d163-4dd8-8a2f-bbf13ab9216f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:00.510344 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac71591-d163-4dd8-8a2f-bbf13ab9216f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:00.511759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac71591-d163-4dd8-8a2f-bbf13ab9216f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:00.532722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 619/1257] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:58:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:00.538986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73665714-5d89-467e-adfb-796f513b7798 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:00.663195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73665714-5d89-467e-adfb-796f513b7798 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:00.706093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9510d395-0f16-441d-ad12-181e9a8c05f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:00.706784 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9510d395-0f16-441d-ad12-181e9a8c05f7 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:00.707061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9510d395-0f16-441d-ad12-181e9a8c05f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:00.707336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9510d395-0f16-441d-ad12-181e9a8c05f7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:00.716779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9510d395-0f16-441d-ad12-181e9a8c05f7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:00.718368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9510d395-0f16-441d-ad12-181e9a8c05f7 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-9510d395-0f16-441d-ad12-181e9a8c05f7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:58:00.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:00.733141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:00.734373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:58:00.734373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:58:00.734373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:58:00.734373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:58:00.734373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:58:00.734373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:58:00.734373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:58:00.734373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:00.734373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 639/1258] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:58:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:00.742008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac71591-d163-4dd8-8a2f-bbf13ab9216f None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:00.859069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac71591-d163-4dd8-8a2f-bbf13ab9216f None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:00.895506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9427f14-f331-46df-a793-6878c549e8ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:00.895991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9427f14-f331-46df-a793-6878c549e8ad None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:00.896273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9427f14-f331-46df-a793-6878c549e8ad None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:00.897104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9427f14-f331-46df-a793-6878c549e8ad None admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:00.897857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9427f14-f331-46df-a793-6878c549e8ad None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:00.898163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9427f14-f331-46df-a793-6878c549e8ad None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:00.899609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9427f14-f331-46df-a793-6878c549e8ad None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:00.914266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 620/1259] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:58:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:00.921132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9510d395-0f16-441d-ad12-181e9a8c05f7 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:00.949560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:01.054639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9510d395-0f16-441d-ad12-181e9a8c05f7 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:01.089323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ddaa05-43d5-4982-a28a-84226bf72d13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:01.089909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ddaa05-43d5-4982-a28a-84226bf72d13 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:01.090185 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ddaa05-43d5-4982-a28a-84226bf72d13 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:01.090465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ddaa05-43d5-4982-a28a-84226bf72d13 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:01.091247 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ddaa05-43d5-4982-a28a-84226bf72d13 None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=1d562ccd50e4492281a9f435f809d49b) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:01.091605 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ddaa05-43d5-4982-a28a-84226bf72d13 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:01.093282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ddaa05-43d5-4982-a28a-84226bf72d13 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:01.100539 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 640/1260] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:58:00 2022] GET /identity/v3/role_assignments?user.id=1d562ccd50e4492281a9f435f809d49b&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:03.132414 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155de5c6-3903-4909-87cd-bf86bf4fbcff None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:03.132950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155de5c6-3903-4909-87cd-bf86bf4fbcff None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:03.133136 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155de5c6-3903-4909-87cd-bf86bf4fbcff None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:03.135208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 621/1261] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:58:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:58:03.141392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ec1534-d678-499e-9da0-f9877bbf8420 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:03.141912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ec1534-d678-499e-9da0-f9877bbf8420 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:03.142350 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ec1534-d678-499e-9da0-f9877bbf8420 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:03.165116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a2ec1534-d678-499e-9da0-f9877bbf8420 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:58:03.166476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a2ec1534-d678-499e-9da0-f9877bbf8420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:58:03.237549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 641/1262] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:58:03 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:58:03.260757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155de5c6-3903-4909-87cd-bf86bf4fbcff None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:03.271878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:03.363033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155de5c6-3903-4909-87cd-bf86bf4fbcff None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:03.399181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdc82c9-67a0-443d-8689-ee0d68c8db27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:03.399763 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdc82c9-67a0-443d-8689-ee0d68c8db27 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:03.400045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdc82c9-67a0-443d-8689-ee0d68c8db27 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:03.400322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdc82c9-67a0-443d-8689-ee0d68c8db27 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:03.411808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdc82c9-67a0-443d-8689-ee0d68c8db27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-5bdc82c9-67a0-443d-8689-ee0d68c8db27 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:58:03.416951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:03.418121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:58:03.419157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 622/1263] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:58:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:03.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ec1534-d678-499e-9da0-f9877bbf8420 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:03.445149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:03.533008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ec1534-d678-499e-9da0-f9877bbf8420 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:03.569092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93db3cd9-af23-484b-9242-d08de8199d7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:03.569707 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93db3cd9-af23-484b-9242-d08de8199d7a None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:03.569991 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93db3cd9-af23-484b-9242-d08de8199d7a None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:03.570265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93db3cd9-af23-484b-9242-d08de8199d7a None admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:03.570977 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93db3cd9-af23-484b-9242-d08de8199d7a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:03.571283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93db3cd9-af23-484b-9242-d08de8199d7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:03.572694 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93db3cd9-af23-484b-9242-d08de8199d7a None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:03.582201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 642/1264] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:58:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:03.588129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdc82c9-67a0-443d-8689-ee0d68c8db27 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:03.655468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:03.690252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdc82c9-67a0-443d-8689-ee0d68c8db27 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:03.723474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa4e412-c57d-4180-8435-0de3a49e0fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:03.724106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa4e412-c57d-4180-8435-0de3a49e0fb5 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:03.724913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa4e412-c57d-4180-8435-0de3a49e0fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:03.725186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa4e412-c57d-4180-8435-0de3a49e0fb5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:03.735559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa4e412-c57d-4180-8435-0de3a49e0fb5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:03.737003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa4e412-c57d-4180-8435-0de3a49e0fb5 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-daa4e412-c57d-4180-8435-0de3a49e0fb5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:58:03.751298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:03.752462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:03.753501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:58:03.753501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:58:03.753501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:58:03.753501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:58:03.753501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 623/1265] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:58:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 31 00:58:03.758930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93db3cd9-af23-484b-9242-d08de8199d7a None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:03.842918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:03.862468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93db3cd9-af23-484b-9242-d08de8199d7a None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:03.896162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9928730d-f39d-4ef8-8926-1ebf1e9dd5a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:03.896770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9928730d-f39d-4ef8-8926-1ebf1e9dd5a2 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:03.897051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9928730d-f39d-4ef8-8926-1ebf1e9dd5a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:03.897322 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9928730d-f39d-4ef8-8926-1ebf1e9dd5a2 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:03.897998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9928730d-f39d-4ef8-8926-1ebf1e9dd5a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:03.898294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9928730d-f39d-4ef8-8926-1ebf1e9dd5a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:03.899675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9928730d-f39d-4ef8-8926-1ebf1e9dd5a2 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:03.911196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 643/1266] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:58:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:03.917537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa4e412-c57d-4180-8435-0de3a49e0fb5 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:04.017726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa4e412-c57d-4180-8435-0de3a49e0fb5 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:04.038008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:04.052338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f9ad4e-d3a2-4acf-b367-56418405ce51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:04.053001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9ad4e-d3a2-4acf-b367-56418405ce51 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:04.053274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9ad4e-d3a2-4acf-b367-56418405ce51 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:04.053539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9ad4e-d3a2-4acf-b367-56418405ce51 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:04.061149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f9ad4e-d3a2-4acf-b367-56418405ce51 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:04.062588 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f9ad4e-d3a2-4acf-b367-56418405ce51 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-54f9ad4e-d3a2-4acf-b367-56418405ce51 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:58:04.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:04.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:04.076085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:04.076085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:04.076085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:58:04.076085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:58:04.076085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:58:04.076085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:58:04.076085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:58:04.076085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:58:04.076085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 31 00:58:04.076085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:58:04.076085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 624/1267] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:58:03 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:04.080603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9928730d-f39d-4ef8-8926-1ebf1e9dd5a2 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:04.189312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9928730d-f39d-4ef8-8926-1ebf1e9dd5a2 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:04.222325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edeccfc-859c-4785-9a74-8a6237f53eca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:04.222924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edeccfc-859c-4785-9a74-8a6237f53eca None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:04.223198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edeccfc-859c-4785-9a74-8a6237f53eca None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:04.223491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edeccfc-859c-4785-9a74-8a6237f53eca None admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:04.224269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edeccfc-859c-4785-9a74-8a6237f53eca None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:04.225226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edeccfc-859c-4785-9a74-8a6237f53eca None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:04.226624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edeccfc-859c-4785-9a74-8a6237f53eca None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:04.235081 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:04.243640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 644/1268] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Jul 31 00:58:04 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:04.249286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f9ad4e-d3a2-4acf-b367-56418405ce51 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:04.350173 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f9ad4e-d3a2-4acf-b367-56418405ce51 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:04.384237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3151bd1-f6d6-4991-9dab-91e38666b6b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:04.384864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3151bd1-f6d6-4991-9dab-91e38666b6b9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:04.385142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3151bd1-f6d6-4991-9dab-91e38666b6b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:04.385417 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3151bd1-f6d6-4991-9dab-91e38666b6b9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:04.395630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3151bd1-f6d6-4991-9dab-91e38666b6b9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:04.397619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3151bd1-f6d6-4991-9dab-91e38666b6b9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-b3151bd1-f6d6-4991-9dab-91e38666b6b9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:58:04.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:04.412467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:04.413468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:58:04.413468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:58:04.413468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:58:04.413468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:58:04.413468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 625/1269] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:58:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:04.418517 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edeccfc-859c-4785-9a74-8a6237f53eca None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:04.518763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edeccfc-859c-4785-9a74-8a6237f53eca None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:04.554391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b430cd-d6f1-4919-a2a9-a460ca2f484f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:04.555005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b430cd-d6f1-4919-a2a9-a460ca2f484f None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:04.555284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b430cd-d6f1-4919-a2a9-a460ca2f484f None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:04.555584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b430cd-d6f1-4919-a2a9-a460ca2f484f None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:04.556272 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b430cd-d6f1-4919-a2a9-a460ca2f484f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:04.556576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b430cd-d6f1-4919-a2a9-a460ca2f484f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:04.557941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b430cd-d6f1-4919-a2a9-a460ca2f484f None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:04.569660 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 645/1270] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:58:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:04.576015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3151bd1-f6d6-4991-9dab-91e38666b6b9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:04.607571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:04.674883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3151bd1-f6d6-4991-9dab-91e38666b6b9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:04.707079 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7926ec6b-fbde-4ba8-8baf-fd420cc6ce99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:04.707762 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7926ec6b-fbde-4ba8-8baf-fd420cc6ce99 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:04.708042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7926ec6b-fbde-4ba8-8baf-fd420cc6ce99 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:04.708315 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7926ec6b-fbde-4ba8-8baf-fd420cc6ce99 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:04.715296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7926ec6b-fbde-4ba8-8baf-fd420cc6ce99 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:04.716753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7926ec6b-fbde-4ba8-8baf-fd420cc6ce99 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-7926ec6b-fbde-4ba8-8baf-fd420cc6ce99 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:58:04.728008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:04.729050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:04.730140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:58:04.730140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:58:04.730140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:58:04.730140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:58:04.730140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:58:04.730140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:58:04.730140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:58:04.730140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:58:04.730140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 626/1271] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:58:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:04.735109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b430cd-d6f1-4919-a2a9-a460ca2f484f None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:04.793060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:04.841237 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b430cd-d6f1-4919-a2a9-a460ca2f484f None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:04.879829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61c39b4-6c63-45de-a241-b0b8ffd60c7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:04.880445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61c39b4-6c63-45de-a241-b0b8ffd60c7c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:04.881354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61c39b4-6c63-45de-a241-b0b8ffd60c7c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:04.881575 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61c39b4-6c63-45de-a241-b0b8ffd60c7c None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:04.882331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61c39b4-6c63-45de-a241-b0b8ffd60c7c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:04.882639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61c39b4-6c63-45de-a241-b0b8ffd60c7c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:04.884108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61c39b4-6c63-45de-a241-b0b8ffd60c7c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:04.896347 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 646/1272] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:58:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:04.902431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7926ec6b-fbde-4ba8-8baf-fd420cc6ce99 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:04.994671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:05.003866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7926ec6b-fbde-4ba8-8baf-fd420cc6ce99 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:05.041559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c39dd2-6c1f-454b-a4de-ca7eebce5e35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:05.042227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c39dd2-6c1f-454b-a4de-ca7eebce5e35 None admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:05.042509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c39dd2-6c1f-454b-a4de-ca7eebce5e35 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:05.042884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c39dd2-6c1f-454b-a4de-ca7eebce5e35 None admin] PATH_INFO: `/v3/projects/8b41573c9f3146d19a306089d686015c/users/1d562ccd50e4492281a9f435f809d49b/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:05.045595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c39dd2-6c1f-454b-a4de-ca7eebce5e35 None admin] RBAC: Authorizing `identity:create_grant(project_id=8b41573c9f3146d19a306089d686015c, user_id=1d562ccd50e4492281a9f435f809d49b, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:05.048563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c39dd2-6c1f-454b-a4de-ca7eebce5e35 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:05.065878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 627/1273] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Jul 31 00:58:04 2022] PUT /identity/v3/projects/8b41573c9f3146d19a306089d686015c/users/1d562ccd50e4492281a9f435f809d49b/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 00:58:07.062742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c62efd-47b4-42a9-9867-0581e06009c7 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:07.063323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c62efd-47b4-42a9-9867-0581e06009c7 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:07.063323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c62efd-47b4-42a9-9867-0581e06009c7 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:07.064825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 647/1274] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:58:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:58:07.070724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1d48be-31ea-4428-ba3b-f76ecfc5b911 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:07.071162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1d48be-31ea-4428-ba3b-f76ecfc5b911 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:07.071274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1d48be-31ea-4428-ba3b-f76ecfc5b911 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:07.089664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ef1d48be-31ea-4428-ba3b-f76ecfc5b911 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:58:07.090856 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ef1d48be-31ea-4428-ba3b-f76ecfc5b911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:58:07.168734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 628/1275] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:58:07 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:58:07.193645 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c62efd-47b4-42a9-9867-0581e06009c7 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:07.294948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c62efd-47b4-42a9-9867-0581e06009c7 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:07.313147 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:07.330864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21348709-34cd-4227-bb7d-0a8d68f0f989 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:07.331528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21348709-34cd-4227-bb7d-0a8d68f0f989 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:07.331813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21348709-34cd-4227-bb7d-0a8d68f0f989 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:07.332085 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21348709-34cd-4227-bb7d-0a8d68f0f989 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:07.341974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21348709-34cd-4227-bb7d-0a8d68f0f989 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-21348709-34cd-4227-bb7d-0a8d68f0f989 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 31 00:58:07.346856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:07.347933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:07.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 648/1276] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Jul 31 00:58:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:07.355133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1d48be-31ea-4428-ba3b-f76ecfc5b911 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:07.433266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:07.455516 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1d48be-31ea-4428-ba3b-f76ecfc5b911 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:07.488749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab1bca6-7e84-464a-b5b2-c7ef3593e48a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:07.489351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab1bca6-7e84-464a-b5b2-c7ef3593e48a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:07.489627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab1bca6-7e84-464a-b5b2-c7ef3593e48a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:07.490465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab1bca6-7e84-464a-b5b2-c7ef3593e48a None admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:07.491163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab1bca6-7e84-464a-b5b2-c7ef3593e48a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:07.491610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab1bca6-7e84-464a-b5b2-c7ef3593e48a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:07.492972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab1bca6-7e84-464a-b5b2-c7ef3593e48a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:07.500977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 629/1277] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Jul 31 00:58:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:07.506738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21348709-34cd-4227-bb7d-0a8d68f0f989 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:07.615661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21348709-34cd-4227-bb7d-0a8d68f0f989 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:07.648149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525e1d1d-fb50-4a75-bc0c-742939dfab62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:07.648786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525e1d1d-fb50-4a75-bc0c-742939dfab62 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:07.649065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525e1d1d-fb50-4a75-bc0c-742939dfab62 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:07.649341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525e1d1d-fb50-4a75-bc0c-742939dfab62 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:07.659678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525e1d1d-fb50-4a75-bc0c-742939dfab62 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:07.661118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525e1d1d-fb50-4a75-bc0c-742939dfab62 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-525e1d1d-fb50-4a75-bc0c-742939dfab62 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:58:07.675906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:07.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:07.678909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:58:07.678909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:58:07.678909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:58:07.678909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:07.678909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 649/1278] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:58:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:07.683439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab1bca6-7e84-464a-b5b2-c7ef3593e48a None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:07.787604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab1bca6-7e84-464a-b5b2-c7ef3593e48a None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:07.823160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f947c2ea-110a-472b-9dfe-2db4d46ffe58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:07.823792 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f947c2ea-110a-472b-9dfe-2db4d46ffe58 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:07.824073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f947c2ea-110a-472b-9dfe-2db4d46ffe58 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:07.824424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f947c2ea-110a-472b-9dfe-2db4d46ffe58 None admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:07.825033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f947c2ea-110a-472b-9dfe-2db4d46ffe58 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:07.825337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f947c2ea-110a-472b-9dfe-2db4d46ffe58 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:07.826696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f947c2ea-110a-472b-9dfe-2db4d46ffe58 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:07.839560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 630/1279] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:58:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:07.845242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525e1d1d-fb50-4a75-bc0c-742939dfab62 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:07.876593 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:07.946854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525e1d1d-fb50-4a75-bc0c-742939dfab62 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:07.979120 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2830a1fc-4235-4fe1-8c1c-b5d73a95fd37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:07.979778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2830a1fc-4235-4fe1-8c1c-b5d73a95fd37 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:07.980052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2830a1fc-4235-4fe1-8c1c-b5d73a95fd37 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:07.980318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2830a1fc-4235-4fe1-8c1c-b5d73a95fd37 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:07.987684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2830a1fc-4235-4fe1-8c1c-b5d73a95fd37 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:07.989111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2830a1fc-4235-4fe1-8c1c-b5d73a95fd37 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-2830a1fc-4235-4fe1-8c1c-b5d73a95fd37 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 31 00:58:08.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:08.001897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:08.003030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:08.003030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:08.003030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 31 00:58:08.003030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 31 00:58:08.003030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 31 00:58:08.003030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 31 00:58:08.003030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 31 00:58:08.003030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 31 00:58:08.003030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 31 00:58:08.003030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:08.003030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 650/1280] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:58:07 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:08.008529 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f947c2ea-110a-472b-9dfe-2db4d46ffe58 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:08.012340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:08.105400 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f947c2ea-110a-472b-9dfe-2db4d46ffe58 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:08.145353 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f649af0e-f098-481c-994d-a2931d7b2f47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:08.145886 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f649af0e-f098-481c-994d-a2931d7b2f47 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:08.146163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f649af0e-f098-481c-994d-a2931d7b2f47 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:08.146470 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f649af0e-f098-481c-994d-a2931d7b2f47 None admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:08.147687 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f649af0e-f098-481c-994d-a2931d7b2f47 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:08.148042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f649af0e-f098-481c-994d-a2931d7b2f47 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:08.149922 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f649af0e-f098-481c-994d-a2931d7b2f47 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:08.168652 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 631/1281] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Jul 31 00:58:08 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:08.174893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2830a1fc-4235-4fe1-8c1c-b5d73a95fd37 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:08.268774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:08.278640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2830a1fc-4235-4fe1-8c1c-b5d73a95fd37 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:08.312564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cca5b2-809c-4991-97c9-ea86c9ef26b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:08.313163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cca5b2-809c-4991-97c9-ea86c9ef26b2 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:08.313447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cca5b2-809c-4991-97c9-ea86c9ef26b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:08.313717 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cca5b2-809c-4991-97c9-ea86c9ef26b2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:08.381628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cca5b2-809c-4991-97c9-ea86c9ef26b2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:08.383068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cca5b2-809c-4991-97c9-ea86c9ef26b2 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:08.471429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:08.482772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 651/1282] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Jul 31 00:58:08 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 31 00:58:08.488734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f649af0e-f098-481c-994d-a2931d7b2f47 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:08.553772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:08.592748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f649af0e-f098-481c-994d-a2931d7b2f47 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:08.627548 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2fc9ec-dfb1-452a-b6c9-43c2ff807ca9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:08.628194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2fc9ec-dfb1-452a-b6c9-43c2ff807ca9 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:08.628469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2fc9ec-dfb1-452a-b6c9-43c2ff807ca9 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:08.628735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2fc9ec-dfb1-452a-b6c9-43c2ff807ca9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:08.639848 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2fc9ec-dfb1-452a-b6c9-43c2ff807ca9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:08.641331 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2fc9ec-dfb1-452a-b6c9-43c2ff807ca9 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-6e2fc9ec-dfb1-452a-b6c9-43c2ff807ca9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 31 00:58:08.657950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:08.659084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:08.660212 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 31 00:58:08.660212 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 31 00:58:08.660212 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 31 00:58:08.660212 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:58:08.660212 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 632/1283] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 00:58:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:08.666050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cca5b2-809c-4991-97c9-ea86c9ef26b2 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:08.777104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cca5b2-809c-4991-97c9-ea86c9ef26b2 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:08.816910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606102a8-2683-440c-820c-19b385cf17b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:08.817528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606102a8-2683-440c-820c-19b385cf17b8 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:08.818349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606102a8-2683-440c-820c-19b385cf17b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:08.818608 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606102a8-2683-440c-820c-19b385cf17b8 None admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:08.819301 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606102a8-2683-440c-820c-19b385cf17b8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:08.819638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606102a8-2683-440c-820c-19b385cf17b8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:08.821033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606102a8-2683-440c-820c-19b385cf17b8 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:08.834940 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 652/1284] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Jul 31 00:58:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:08.840895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2fc9ec-dfb1-452a-b6c9-43c2ff807ca9 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:08.940862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2fc9ec-dfb1-452a-b6c9-43c2ff807ca9 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:08.955072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:08.976063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846ed31b-c09d-4372-a3f9-3b9b962a2743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:08.976696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846ed31b-c09d-4372-a3f9-3b9b962a2743 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:08.976977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846ed31b-c09d-4372-a3f9-3b9b962a2743 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:08.977253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846ed31b-c09d-4372-a3f9-3b9b962a2743 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:08.983901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846ed31b-c09d-4372-a3f9-3b9b962a2743 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:08.985343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846ed31b-c09d-4372-a3f9-3b9b962a2743 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application [None req-846ed31b-c09d-4372-a3f9-3b9b962a2743 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 31 00:58:08.996666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:08.997727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:08.998697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 31 00:58:08.998697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 31 00:58:08.998697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 31 00:58:08.998697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 31 00:58:08.998697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 31 00:58:08.998697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 31 00:58:08.998697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 31 00:58:08.998697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: ERROR keystone.server.flask.application Jul 31 00:58:08.998697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 633/1285] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Jul 31 00:58:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:09.003705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606102a8-2683-440c-820c-19b385cf17b8 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:09.053314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:09.105155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606102a8-2683-440c-820c-19b385cf17b8 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:09.143821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccaf046-a9ee-4449-9e1d-20b6e615efe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:09.144449 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccaf046-a9ee-4449-9e1d-20b6e615efe5 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:09.144728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccaf046-a9ee-4449-9e1d-20b6e615efe5 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:09.144997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccaf046-a9ee-4449-9e1d-20b6e615efe5 None admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:09.145757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccaf046-a9ee-4449-9e1d-20b6e615efe5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:09.146714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccaf046-a9ee-4449-9e1d-20b6e615efe5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:09.148173 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccaf046-a9ee-4449-9e1d-20b6e615efe5 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:09.160524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 653/1286] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Jul 31 00:58:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:09.172103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846ed31b-c09d-4372-a3f9-3b9b962a2743 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:09.272159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846ed31b-c09d-4372-a3f9-3b9b962a2743 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:09.305097 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53417074-8cf3-49ae-877a-0f649381471f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:09.305623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53417074-8cf3-49ae-877a-0f649381471f None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:09.305903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53417074-8cf3-49ae-877a-0f649381471f None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:09.306175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53417074-8cf3-49ae-877a-0f649381471f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:09.306953 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53417074-8cf3-49ae-877a-0f649381471f None admin] RBAC: Adding query filter params (role.id=8814964206e14dc3ae6ab69e8d874f51, scope.project.id=8b41573c9f3146d19a306089d686015c, user.id=1d562ccd50e4492281a9f435f809d49b) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:09.307260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53417074-8cf3-49ae-877a-0f649381471f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:09.308718 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53417074-8cf3-49ae-877a-0f649381471f None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:09.314473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 634/1287] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Jul 31 00:58:09 2022] GET /identity/v3/role_assignments?user.id=1d562ccd50e4492281a9f435f809d49b&scope.project.id=8b41573c9f3146d19a306089d686015c&role.id=8814964206e14dc3ae6ab69e8d874f51 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 00:58:11.353554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855be139-90e5-4626-ab73-ce102fdb0035 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:11.354163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855be139-90e5-4626-ab73-ce102fdb0035 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:11.354163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855be139-90e5-4626-ab73-ce102fdb0035 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:11.355488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 654/1288] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:58:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:58:11.361332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24df26ba-c05f-408a-8778-89e4dba561f1 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:11.361868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24df26ba-c05f-408a-8778-89e4dba561f1 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:11.361868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24df26ba-c05f-408a-8778-89e4dba561f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:11.382216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-24df26ba-c05f-408a-8778-89e4dba561f1 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:58:11.383491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-24df26ba-c05f-408a-8778-89e4dba561f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:58:11.416926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:11.459620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 635/1289] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:58:11 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:58:11.481604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855be139-90e5-4626-ab73-ce102fdb0035 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:11.585509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855be139-90e5-4626-ab73-ce102fdb0035 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:11.596228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:11.619667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cb3974-d6a2-42c9-bdf2-d69d3a110eda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:11.620218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cb3974-d6a2-42c9-bdf2-d69d3a110eda None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:11.620496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cb3974-d6a2-42c9-bdf2-d69d3a110eda None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:11.620768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cb3974-d6a2-42c9-bdf2-d69d3a110eda None admin] PATH_INFO: `/v3/services/placement` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:11.621393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cb3974-d6a2-42c9-bdf2-d69d3a110eda None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:11.622753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cb3974-d6a2-42c9-bdf2-d69d3a110eda None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-e9cb3974-d6a2-42c9-bdf2-d69d3a110eda None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:11.632186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:11.633388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:11.634346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 655/1290] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Jul 31 00:58:11 2022] GET /identity/v3/services/placement => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:11.639401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24df26ba-c05f-408a-8778-89e4dba561f1 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:11.736653 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24df26ba-c05f-408a-8778-89e4dba561f1 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:11.770162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de92ded-16a4-4c73-ba9a-585589e217ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:11.770766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de92ded-16a4-4c73-ba9a-585589e217ad None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:11.771040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de92ded-16a4-4c73-ba9a-585589e217ad None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:11.771309 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de92ded-16a4-4c73-ba9a-585589e217ad None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:11.772004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de92ded-16a4-4c73-ba9a-585589e217ad None admin] RBAC: Adding query filter params (name=placement) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:11.772301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de92ded-16a4-4c73-ba9a-585589e217ad None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:11.773673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de92ded-16a4-4c73-ba9a-585589e217ad None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:11.778580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 636/1291] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Jul 31 00:58:11 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:11.784464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cb3974-d6a2-42c9-bdf2-d69d3a110eda None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:11.893354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cb3974-d6a2-42c9-bdf2-d69d3a110eda None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:11.926855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6b352e-1f3a-4646-acb5-ee84a779ce21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:11.928106 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6b352e-1f3a-4646-acb5-ee84a779ce21 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:11.928377 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6b352e-1f3a-4646-acb5-ee84a779ce21 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:11.928643 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6b352e-1f3a-4646-acb5-ee84a779ce21 None admin] PATH_INFO: `/v3/services` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:11.929307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6b352e-1f3a-4646-acb5-ee84a779ce21 None admin] RBAC: Adding query filter params (type=placement) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:11.929604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6b352e-1f3a-4646-acb5-ee84a779ce21 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:11.930956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6b352e-1f3a-4646-acb5-ee84a779ce21 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:11.935234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 656/1292] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Jul 31 00:58:11 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:13.919258 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f39e674-0509-4f10-a70c-5d3916fbc52a None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:13.919890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f39e674-0509-4f10-a70c-5d3916fbc52a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:13.919890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f39e674-0509-4f10-a70c-5d3916fbc52a None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:13.921431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 637/1293] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:58:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:58:13.926398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfd0267-130a-4780-ab89-a8e740a8ab1f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:13.926871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfd0267-130a-4780-ab89-a8e740a8ab1f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:13.926982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfd0267-130a-4780-ab89-a8e740a8ab1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:13.944793 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ddfd0267-130a-4780-ab89-a8e740a8ab1f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:58:13.945969 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ddfd0267-130a-4780-ab89-a8e740a8ab1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:58:13.975502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:14.013570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 657/1294] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:58:13 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:58:14.035597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f39e674-0509-4f10-a70c-5d3916fbc52a None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:14.066490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:14.140705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f39e674-0509-4f10-a70c-5d3916fbc52a None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:14.181267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be774070-f7b1-4ab3-a820-d7371721d886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:14.181859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be774070-f7b1-4ab3-a820-d7371721d886 None admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:14.182137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be774070-f7b1-4ab3-a820-d7371721d886 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:14.182498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be774070-f7b1-4ab3-a820-d7371721d886 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:14.183329 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be774070-f7b1-4ab3-a820-d7371721d886 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:14.185021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be774070-f7b1-4ab3-a820-d7371721d886 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:14.202297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 638/1295] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Jul 31 00:58:14 2022] POST /identity/v3/services => generated 246 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:58:16.213569 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db4490a-cb8d-4695-bd0d-bedd2f063db7 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:16.214129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db4490a-cb8d-4695-bd0d-bedd2f063db7 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:16.214129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db4490a-cb8d-4695-bd0d-bedd2f063db7 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:16.215487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 658/1296] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:58:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:58:16.220826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34880b43-f74a-46b6-ae83-7eba550d9935 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:16.221313 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34880b43-f74a-46b6-ae83-7eba550d9935 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:16.221410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34880b43-f74a-46b6-ae83-7eba550d9935 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:16.239711 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-34880b43-f74a-46b6-ae83-7eba550d9935 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:58:16.240979 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-34880b43-f74a-46b6-ae83-7eba550d9935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:58:16.313569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 639/1297] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:58:16 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:58:16.336822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db4490a-cb8d-4695-bd0d-bedd2f063db7 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:16.438244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db4490a-cb8d-4695-bd0d-bedd2f063db7 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:16.472614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8548057c-bf8e-490e-8f61-71ec510a72df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:16.473188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8548057c-bf8e-490e-8f61-71ec510a72df None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:16.473486 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8548057c-bf8e-490e-8f61-71ec510a72df None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:16.474286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8548057c-bf8e-490e-8f61-71ec510a72df None admin] PATH_INFO: `/v3/services/placement` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:16.474912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8548057c-bf8e-490e-8f61-71ec510a72df None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:16.476323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8548057c-bf8e-490e-8f61-71ec510a72df None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-8548057c-bf8e-490e-8f61-71ec510a72df None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:16.485260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:16.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 31 00:58:16.487623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:16.491548 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 659/1298] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Jul 31 00:58:16 2022] GET /identity/v3/services/placement => generated 90 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:16.493443 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34880b43-f74a-46b6-ae83-7eba550d9935 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:16.511900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:16.596576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34880b43-f74a-46b6-ae83-7eba550d9935 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:16.630847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04d7114-88c3-4dad-bfc2-ae06c5a53756 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:16.631491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04d7114-88c3-4dad-bfc2-ae06c5a53756 None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:16.631769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04d7114-88c3-4dad-bfc2-ae06c5a53756 None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:16.632040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04d7114-88c3-4dad-bfc2-ae06c5a53756 None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:16.632708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04d7114-88c3-4dad-bfc2-ae06c5a53756 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:16.633012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04d7114-88c3-4dad-bfc2-ae06c5a53756 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:16.634383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04d7114-88c3-4dad-bfc2-ae06c5a53756 None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:16.639671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 640/1299] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Jul 31 00:58:16 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:16.645762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8548057c-bf8e-490e-8f61-71ec510a72df None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:16.679249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:16.747299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8548057c-bf8e-490e-8f61-71ec510a72df None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:16.780601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c97157-8722-406c-813b-eefd9aeb785c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:16.781214 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c97157-8722-406c-813b-eefd9aeb785c None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:16.781487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c97157-8722-406c-813b-eefd9aeb785c None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:16.781754 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c97157-8722-406c-813b-eefd9aeb785c None admin] PATH_INFO: `/v3/endpoints` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:16.782504 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c97157-8722-406c-813b-eefd9aeb785c None admin] RBAC: Adding query filter params (interface=public, service_id=5f4a50d666a54f65abf7b1b79129c43d, region_id=RegionOne) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:16.782808 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c97157-8722-406c-813b-eefd9aeb785c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:16.784202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c97157-8722-406c-813b-eefd9aeb785c None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:16.789951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 660/1300] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Jul 31 00:58:16 2022] GET /identity/v3/endpoints?service_id=5f4a50d666a54f65abf7b1b79129c43d&interface=public®ion_id=RegionOne => generated 194 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:16.795657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04d7114-88c3-4dad-bfc2-ae06c5a53756 None admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:16.877698 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:16.897236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04d7114-88c3-4dad-bfc2-ae06c5a53756 None admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:16.931783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e500afd1-7380-4246-861e-7eb2f388657a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:16.932403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e500afd1-7380-4246-861e-7eb2f388657a None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:16.932682 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e500afd1-7380-4246-861e-7eb2f388657a None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:16.932956 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e500afd1-7380-4246-861e-7eb2f388657a None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:16.933607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e500afd1-7380-4246-861e-7eb2f388657a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:16.934986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e500afd1-7380-4246-861e-7eb2f388657a None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:16.942190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 641/1301] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Jul 31 00:58:16 2022] GET /identity/v3/services => generated 2191 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 00:58:18.936077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93aff4e-623a-46be-a032-9251e22b651a None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:18.936613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93aff4e-623a-46be-a032-9251e22b651a None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:18.936613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93aff4e-623a-46be-a032-9251e22b651a None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:18.937986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 661/1302] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:58:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:58:18.943746 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc02aa03-a09b-4ac7-be05-1d0891afe97d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:18.944163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc02aa03-a09b-4ac7-be05-1d0891afe97d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:18.944295 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc02aa03-a09b-4ac7-be05-1d0891afe97d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:18.962519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-dc02aa03-a09b-4ac7-be05-1d0891afe97d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:58:18.963740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-dc02aa03-a09b-4ac7-be05-1d0891afe97d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:58:19.028973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 642/1303] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:58:18 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 31 00:58:19.051372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93aff4e-623a-46be-a032-9251e22b651a None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:19.151437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:19.159826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93aff4e-623a-46be-a032-9251e22b651a None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:19.198471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f99bd16-7660-4036-a418-c50b1f2d4316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:19.199180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f99bd16-7660-4036-a418-c50b1f2d4316 None admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:19.199741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f99bd16-7660-4036-a418-c50b1f2d4316 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:19.200133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f99bd16-7660-4036-a418-c50b1f2d4316 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:19.200776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f99bd16-7660-4036-a418-c50b1f2d4316 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:19.202179 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f99bd16-7660-4036-a418-c50b1f2d4316 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application [None req-2f99bd16-7660-4036-a418-c50b1f2d4316 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 31 00:58:19.211794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 31 00:58:19.213086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.get_or_create( Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application with Lock( Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self._enter() Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application return self.creator() Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application created_value = creator( Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: ERROR keystone.server.flask.application Jul 31 00:58:19.214312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 662/1304] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Jul 31 00:58:19 2022] GET /identity/v3/services/placement => generated 90 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 31 00:58:19.219008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc02aa03-a09b-4ac7-be05-1d0891afe97d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:19.293992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:19.318073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc02aa03-a09b-4ac7-be05-1d0891afe97d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:19.351716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0875cd01-7d2a-4980-a244-c7ccb6d276db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:19.352323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0875cd01-7d2a-4980-a244-c7ccb6d276db None admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:19.352598 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0875cd01-7d2a-4980-a244-c7ccb6d276db None admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:19.352869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0875cd01-7d2a-4980-a244-c7ccb6d276db None admin] PATH_INFO: `/v3/services` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:19.353529 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0875cd01-7d2a-4980-a244-c7ccb6d276db None admin] RBAC: Adding query filter params (name=placement) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 00:58:19.353828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0875cd01-7d2a-4980-a244-c7ccb6d276db None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:19.355186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0875cd01-7d2a-4980-a244-c7ccb6d276db None admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:19.360216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 643/1305] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Jul 31 00:58:19 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 00:58:19.366576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f99bd16-7660-4036-a418-c50b1f2d4316 None admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:58:19.464349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f99bd16-7660-4036-a418-c50b1f2d4316 None admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:58:19.498613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebe6445-6d09-4cc0-827d-918997148046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:58:19.499224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe6445-6d09-4cc0-827d-918997148046 None admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:58:19.499536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe6445-6d09-4cc0-827d-918997148046 None admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:58:19.499810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe6445-6d09-4cc0-827d-918997148046 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:58:19.500527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebe6445-6d09-4cc0-827d-918997148046 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:58:19.501881 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebe6445-6d09-4cc0-827d-918997148046 None admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:58:19.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 663/1306] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 00:58:19 2022] POST /identity/v3/endpoints => generated 331 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 00:58:29.357307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=30 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:58:29.525619 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:23.899235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31822ac-65db-4770-aff5-d15e9f318ff9 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:23.899743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31822ac-65db-4770-aff5-d15e9f318ff9 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:23.899995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31822ac-65db-4770-aff5-d15e9f318ff9 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:23.905321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 644/1307] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:59:23 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:23.906611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a774c699-90ef-44df-a9de-98634c8e1d7e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:23.907083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a774c699-90ef-44df-a9de-98634c8e1d7e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:23.907269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a774c699-90ef-44df-a9de-98634c8e1d7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:24.006479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a774c699-90ef-44df-a9de-98634c8e1d7e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:24.007931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a774c699-90ef-44df-a9de-98634c8e1d7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:24.139448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 664/1308] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:59:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:24.188869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6830eb5-d76f-4d34-b54a-1464a58c3e49 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:24.189142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6830eb5-d76f-4d34-b54a-1464a58c3e49 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:24.189668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6830eb5-d76f-4d34-b54a-1464a58c3e49 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:24.191112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 645/1309] 10.222.0.42 () {54 vars in 965 bytes} [Sun Jul 31 00:59:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:24.197068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd603681-e210-435d-882a-1d3b5b30f524 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:24.197355 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd603681-e210-435d-882a-1d3b5b30f524 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:24.197629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd603681-e210-435d-882a-1d3b5b30f524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:24.278772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-bd603681-e210-435d-882a-1d3b5b30f524 None None] MFA Rules not processed for user `94e868131d33472b9ded8486274a07b9`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:24.280003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-bd603681-e210-435d-882a-1d3b5b30f524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:24.302002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:24.361946 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 665/1310] 10.222.0.42 () {58 vars in 1062 bytes} [Sun Jul 31 00:59:24 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:24.371022 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6830eb5-d76f-4d34-b54a-1464a58c3e49 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:59:24.372756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b6830eb5-d76f-4d34-b54a-1464a58c3e49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:24.498232 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6830eb5-d76f-4d34-b54a-1464a58c3e49 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:59:24.593394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64a8b2c-5333-4c5e-8adc-2b0f6e91bff7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:59:24.594027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64a8b2c-5333-4c5e-8adc-2b0f6e91bff7 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:24.594332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64a8b2c-5333-4c5e-8adc-2b0f6e91bff7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:24.594605 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64a8b2c-5333-4c5e-8adc-2b0f6e91bff7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:24.597628 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a64a8b2c-5333-4c5e-8adc-2b0f6e91bff7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:24.662851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:24.725190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64a8b2c-5333-4c5e-8adc-2b0f6e91bff7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:59:24.726629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64a8b2c-5333-4c5e-8adc-2b0f6e91bff7 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:59:24.747501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 646/1311] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 00:59:24 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 00:59:27.085730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcc0a58-03f3-47ca-8138-d2e5ef362fff None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:27.086258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcc0a58-03f3-47ca-8138-d2e5ef362fff None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:27.086258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcc0a58-03f3-47ca-8138-d2e5ef362fff None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:27.087648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 666/1312] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:59:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:27.093208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a8766d-c068-418f-88f7-6791cf6a0942 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:27.093632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a8766d-c068-418f-88f7-6791cf6a0942 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:27.093752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a8766d-c068-418f-88f7-6791cf6a0942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:27.174477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-68a8766d-c068-418f-88f7-6791cf6a0942 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:27.175752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-68a8766d-c068-418f-88f7-6791cf6a0942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:27.278861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 647/1313] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:59:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:27.340224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc010b6c-09d7-4e36-abbb-b5a7689278a5 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:27.340510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc010b6c-09d7-4e36-abbb-b5a7689278a5 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:27.340796 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc010b6c-09d7-4e36-abbb-b5a7689278a5 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:27.342158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 667/1314] 10.222.0.42 () {54 vars in 965 bytes} [Sun Jul 31 00:59:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:27.349170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dfaf01-e008-4786-b0c1-41f5eed3c47d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:27.349457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dfaf01-e008-4786-b0c1-41f5eed3c47d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:27.349734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dfaf01-e008-4786-b0c1-41f5eed3c47d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:27.361809 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:27.428259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d5dfaf01-e008-4786-b0c1-41f5eed3c47d None None] MFA Rules not processed for user `94e868131d33472b9ded8486274a07b9`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:27.429485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d5dfaf01-e008-4786-b0c1-41f5eed3c47d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:27.506427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 648/1315] 10.222.0.42 () {58 vars in 1062 bytes} [Sun Jul 31 00:59:27 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:27.515720 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc010b6c-09d7-4e36-abbb-b5a7689278a5 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:59:27.517824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-dc010b6c-09d7-4e36-abbb-b5a7689278a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:27.626201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc010b6c-09d7-4e36-abbb-b5a7689278a5 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:59:27.696154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:27.699384 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2923f49f-9355-492d-8b99-e07d9aa4561c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:59:27.700030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2923f49f-9355-492d-8b99-e07d9aa4561c service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:27.700518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2923f49f-9355-492d-8b99-e07d9aa4561c service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:27.700993 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2923f49f-9355-492d-8b99-e07d9aa4561c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:27.702692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2923f49f-9355-492d-8b99-e07d9aa4561c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:27.811497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2923f49f-9355-492d-8b99-e07d9aa4561c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:59:27.813501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2923f49f-9355-492d-8b99-e07d9aa4561c service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:59:27.834036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 668/1316] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 00:59:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 00:59:30.382312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab71886-3d42-4d78-907c-079bf06205ef None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:30.382825 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab71886-3d42-4d78-907c-079bf06205ef None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:30.382825 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab71886-3d42-4d78-907c-079bf06205ef None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:30.384143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 649/1317] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:59:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:30.389247 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01c54be-2043-4b38-be18-0b67c0623666 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:30.389720 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01c54be-2043-4b38-be18-0b67c0623666 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:30.389827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01c54be-2043-4b38-be18-0b67c0623666 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:30.456098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b01c54be-2043-4b38-be18-0b67c0623666 None None] MFA Rules not processed for user `a614cb9f929748ecb70ea9ce0aed9916`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:30.457308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b01c54be-2043-4b38-be18-0b67c0623666 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:30.510079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:30.558076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 669/1318] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:59:30 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:30.599062 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab71886-3d42-4d78-907c-079bf06205ef None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:59:30.600634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-cab71886-3d42-4d78-907c-079bf06205ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:30.708865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab71886-3d42-4d78-907c-079bf06205ef None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:59:30.711734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:30.782497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2bd687-3dc9-4811-8672-cebd662e11e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:59:30.783105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2bd687-3dc9-4811-8672-cebd662e11e8 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:30.783405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2bd687-3dc9-4811-8672-cebd662e11e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:30.783702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2bd687-3dc9-4811-8672-cebd662e11e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:30.785319 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6d2bd687-3dc9-4811-8672-cebd662e11e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:30.893879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2bd687-3dc9-4811-8672-cebd662e11e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:59:30.895330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2bd687-3dc9-4811-8672-cebd662e11e8 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:59:30.916256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 650/1319] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 00:59:30 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 00:59:33.550729 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211c4e6c-b3ba-4246-8bba-ff5881a64dee None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:33.551196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211c4e6c-b3ba-4246-8bba-ff5881a64dee None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:33.551310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211c4e6c-b3ba-4246-8bba-ff5881a64dee None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:33.552719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 670/1320] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:59:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:33.558199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adba724-9104-490a-aa02-48dd3210f8a7 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:33.558646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adba724-9104-490a-aa02-48dd3210f8a7 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:33.558737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adba724-9104-490a-aa02-48dd3210f8a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:33.601351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:33.629422 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-7adba724-9104-490a-aa02-48dd3210f8a7 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:33.630661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7adba724-9104-490a-aa02-48dd3210f8a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:33.726959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 651/1321] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:59:33 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:33.766040 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211c4e6c-b3ba-4246-8bba-ff5881a64dee None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:59:33.767646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-211c4e6c-b3ba-4246-8bba-ff5881a64dee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:33.873469 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211c4e6c-b3ba-4246-8bba-ff5881a64dee None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:59:33.946128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c626d02-7ed0-433e-9f33-a053098ab38e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:59:33.946742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c626d02-7ed0-433e-9f33-a053098ab38e service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:33.947019 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c626d02-7ed0-433e-9f33-a053098ab38e service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:33.947287 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c626d02-7ed0-433e-9f33-a053098ab38e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:33.948932 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6c626d02-7ed0-433e-9f33-a053098ab38e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:33.957041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:34.058769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c626d02-7ed0-433e-9f33-a053098ab38e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:59:34.060233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c626d02-7ed0-433e-9f33-a053098ab38e service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:59:34.080186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 671/1322] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 00:59:33 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 00:59:36.779805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa81827-68a4-4ab8-943d-23f3514b0798 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:36.780366 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa81827-68a4-4ab8-943d-23f3514b0798 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:36.780366 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa81827-68a4-4ab8-943d-23f3514b0798 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:36.782299 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 652/1323] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:59:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:36.787992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eefdd8f-0cc2-4caf-8d8a-5ff1079ad15b None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:36.788433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eefdd8f-0cc2-4caf-8d8a-5ff1079ad15b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:36.788550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eefdd8f-0cc2-4caf-8d8a-5ff1079ad15b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:36.855438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-3eefdd8f-0cc2-4caf-8d8a-5ff1079ad15b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:36.856577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3eefdd8f-0cc2-4caf-8d8a-5ff1079ad15b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:36.915125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:36.960169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 672/1324] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:59:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:36.997080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa81827-68a4-4ab8-943d-23f3514b0798 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:59:36.998726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7aa81827-68a4-4ab8-943d-23f3514b0798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:37.121513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa81827-68a4-4ab8-943d-23f3514b0798 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:59:37.136163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:37.205555 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948d3270-9fc3-4dc9-b812-a897750851f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:59:37.206167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948d3270-9fc3-4dc9-b812-a897750851f8 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:37.206445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948d3270-9fc3-4dc9-b812-a897750851f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:37.206714 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948d3270-9fc3-4dc9-b812-a897750851f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:37.208383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-948d3270-9fc3-4dc9-b812-a897750851f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:37.325851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948d3270-9fc3-4dc9-b812-a897750851f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:59:37.328488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948d3270-9fc3-4dc9-b812-a897750851f8 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:59:37.348525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 653/1325] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 00:59:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 00:59:39.422428 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230b122e-070b-4d84-810c-b3312b5eaaf3 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:39.423221 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230b122e-070b-4d84-810c-b3312b5eaaf3 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:39.423221 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230b122e-070b-4d84-810c-b3312b5eaaf3 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:39.424338 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 673/1326] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:59:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:39.429518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441309eb-43e1-4c59-b303-3b1f8669264c None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:39.429959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441309eb-43e1-4c59-b303-3b1f8669264c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:39.430053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441309eb-43e1-4c59-b303-3b1f8669264c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:39.480008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:39.498738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-441309eb-43e1-4c59-b303-3b1f8669264c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:39.499993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-441309eb-43e1-4c59-b303-3b1f8669264c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:39.606357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 654/1327] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:59:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:39.643152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230b122e-070b-4d84-810c-b3312b5eaaf3 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:59:39.644738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-230b122e-070b-4d84-810c-b3312b5eaaf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:39.748599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230b122e-070b-4d84-810c-b3312b5eaaf3 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:59:39.823596 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96d06e3-c726-40a1-82c9-4b96d2eede58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:59:39.824212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96d06e3-c726-40a1-82c9-4b96d2eede58 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:39.824485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96d06e3-c726-40a1-82c9-4b96d2eede58 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:39.824749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96d06e3-c726-40a1-82c9-4b96d2eede58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:39.826388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a96d06e3-c726-40a1-82c9-4b96d2eede58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:39.844555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:39.941635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96d06e3-c726-40a1-82c9-4b96d2eede58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:59:39.943072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96d06e3-c726-40a1-82c9-4b96d2eede58 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:59:39.961788 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 674/1328] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 00:59:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 00:59:42.054966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf3451f-55fc-4201-879f-7f865d310461 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:42.055528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf3451f-55fc-4201-879f-7f865d310461 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:42.055528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf3451f-55fc-4201-879f-7f865d310461 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:42.056911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 655/1329] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:59:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:42.062703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3320fb72-928a-44e4-8197-3b72b0acf869 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:42.063129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3320fb72-928a-44e4-8197-3b72b0acf869 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:42.063250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3320fb72-928a-44e4-8197-3b72b0acf869 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:42.134843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-3320fb72-928a-44e4-8197-3b72b0acf869 None None] MFA Rules not processed for user `a614cb9f929748ecb70ea9ce0aed9916`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:42.136096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3320fb72-928a-44e4-8197-3b72b0acf869 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:42.204917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:42.243026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 675/1330] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:59:42 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:42.280352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf3451f-55fc-4201-879f-7f865d310461 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:59:42.281871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ddf3451f-55fc-4201-879f-7f865d310461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:42.390135 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf3451f-55fc-4201-879f-7f865d310461 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:59:42.409862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:42.465078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d663c23-9a12-4b43-8417-1e5c04d4772c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:59:42.465614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d663c23-9a12-4b43-8417-1e5c04d4772c service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:42.465892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d663c23-9a12-4b43-8417-1e5c04d4772c service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:42.466165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d663c23-9a12-4b43-8417-1e5c04d4772c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:42.467784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-3d663c23-9a12-4b43-8417-1e5c04d4772c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:42.582357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d663c23-9a12-4b43-8417-1e5c04d4772c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:59:42.583830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d663c23-9a12-4b43-8417-1e5c04d4772c service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:59:42.602697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 656/1331] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 00:59:42 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 00:59:44.880924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12161516-d329-4372-87ab-9d756677a3b5 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:44.881454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12161516-d329-4372-87ab-9d756677a3b5 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:44.881454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12161516-d329-4372-87ab-9d756677a3b5 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:44.882753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 676/1332] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:59:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:44.888104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f89f50c-bb5c-485b-b425-4c91c987cea9 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:44.888562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f89f50c-bb5c-485b-b425-4c91c987cea9 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:44.888650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f89f50c-bb5c-485b-b425-4c91c987cea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:44.951966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:44.956143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6f89f50c-bb5c-485b-b425-4c91c987cea9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:44.957539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6f89f50c-bb5c-485b-b425-4c91c987cea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:45.053854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 657/1333] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:59:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:45.090528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12161516-d329-4372-87ab-9d756677a3b5 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:59:45.092083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-12161516-d329-4372-87ab-9d756677a3b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:45.194684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12161516-d329-4372-87ab-9d756677a3b5 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:59:45.268249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb35b7d-b8a8-4759-85fc-49589170efde service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:59:45.268860 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb35b7d-b8a8-4759-85fc-49589170efde service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:45.269132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb35b7d-b8a8-4759-85fc-49589170efde service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:45.269397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb35b7d-b8a8-4759-85fc-49589170efde service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:45.271100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-feb35b7d-b8a8-4759-85fc-49589170efde service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:45.298013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:45.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb35b7d-b8a8-4759-85fc-49589170efde service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:59:45.385074 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb35b7d-b8a8-4759-85fc-49589170efde service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:59:45.403592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 677/1334] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 00:59:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 00:59:47.482387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a355d2-51ad-4592-bca1-ab1b7e5af97b None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:47.482943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a355d2-51ad-4592-bca1-ab1b7e5af97b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:47.482943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a355d2-51ad-4592-bca1-ab1b7e5af97b None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:47.484860 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 658/1335] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:59:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:47.489978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51131a7-bb2f-4ff3-ba64-04711cc5899b None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:47.490408 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51131a7-bb2f-4ff3-ba64-04711cc5899b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:47.490535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51131a7-bb2f-4ff3-ba64-04711cc5899b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:47.558907 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-d51131a7-bb2f-4ff3-ba64-04711cc5899b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:47.560253 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d51131a7-bb2f-4ff3-ba64-04711cc5899b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:47.637601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:47.661040 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 678/1336] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:59:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:47.702032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a355d2-51ad-4592-bca1-ab1b7e5af97b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:59:47.703601 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-45a355d2-51ad-4592-bca1-ab1b7e5af97b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:47.807369 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a355d2-51ad-4592-bca1-ab1b7e5af97b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:59:47.835669 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:47.880787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a29c0f-77ad-4c6f-b32e-df3665fe1370 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:59:47.881394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a29c0f-77ad-4c6f-b32e-df3665fe1370 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:47.881671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a29c0f-77ad-4c6f-b32e-df3665fe1370 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:47.881957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a29c0f-77ad-4c6f-b32e-df3665fe1370 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:47.883556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c6a29c0f-77ad-4c6f-b32e-df3665fe1370 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:47.991237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a29c0f-77ad-4c6f-b32e-df3665fe1370 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:59:47.992679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a29c0f-77ad-4c6f-b32e-df3665fe1370 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:59:48.010080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 659/1337] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 00:59:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 00:59:50.364023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeb7a97-39f8-416b-ab6a-0440c276d04f None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:50.364564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeb7a97-39f8-416b-ab6a-0440c276d04f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:50.364564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeb7a97-39f8-416b-ab6a-0440c276d04f None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:50.365930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 679/1338] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:59:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:50.371092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c083de-b317-4a36-b32d-4d600550541d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:50.371523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c083de-b317-4a36-b32d-4d600550541d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:50.371661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c083de-b317-4a36-b32d-4d600550541d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:50.436863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-00c083de-b317-4a36-b32d-4d600550541d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:50.438043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-00c083de-b317-4a36-b32d-4d600550541d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:50.446122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:50.537649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 660/1339] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:59:50 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:50.574517 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eeb7a97-39f8-416b-ab6a-0440c276d04f None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:59:50.576069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2eeb7a97-39f8-416b-ab6a-0440c276d04f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:50.678462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eeb7a97-39f8-416b-ab6a-0440c276d04f None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:59:50.751673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76ba0c6-a5b5-4569-b7bd-32675f74972b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:59:50.752289 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76ba0c6-a5b5-4569-b7bd-32675f74972b service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:50.752567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76ba0c6-a5b5-4569-b7bd-32675f74972b service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:50.752838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76ba0c6-a5b5-4569-b7bd-32675f74972b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:50.754448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f76ba0c6-a5b5-4569-b7bd-32675f74972b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:50.788704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:50.862421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76ba0c6-a5b5-4569-b7bd-32675f74972b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:59:50.863894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76ba0c6-a5b5-4569-b7bd-32675f74972b service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:59:50.881590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 680/1340] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 00:59:50 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 00:59:54.813964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c69917-ae05-4c91-b66b-714b073cceba None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:54.814478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c69917-ae05-4c91-b66b-714b073cceba None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:54.814478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c69917-ae05-4c91-b66b-714b073cceba None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:54.815826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 661/1341] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:59:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:54.821093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf6be43-e94a-434f-ad49-ffdcc6df1722 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:54.821554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf6be43-e94a-434f-ad49-ffdcc6df1722 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:54.821638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf6be43-e94a-434f-ad49-ffdcc6df1722 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:54.889243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-4cf6be43-e94a-434f-ad49-ffdcc6df1722 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:54.890443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4cf6be43-e94a-434f-ad49-ffdcc6df1722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:54.980296 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:54.992146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 681/1342] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:59:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:55.031340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c69917-ae05-4c91-b66b-714b073cceba None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:59:55.032915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-30c69917-ae05-4c91-b66b-714b073cceba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:55.146111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c69917-ae05-4c91-b66b-714b073cceba None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:59:55.186002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:55.223547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6233d3-7de4-4f26-8f03-778dd9b92985 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:59:55.224164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6233d3-7de4-4f26-8f03-778dd9b92985 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:55.224445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6233d3-7de4-4f26-8f03-778dd9b92985 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:55.224719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6233d3-7de4-4f26-8f03-778dd9b92985 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:55.226348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1d6233d3-7de4-4f26-8f03-778dd9b92985 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:55.339595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6233d3-7de4-4f26-8f03-778dd9b92985 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:59:55.341035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6233d3-7de4-4f26-8f03-778dd9b92985 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:59:55.360597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 662/1343] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 00:59:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 00:59:58.081933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584bb401-ac6a-43fc-94d0-a7efe583d21e None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:58.082639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584bb401-ac6a-43fc-94d0-a7efe583d21e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:58.082639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584bb401-ac6a-43fc-94d0-a7efe583d21e None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:58.083987 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 682/1344] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 00:59:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 00:59:58.089964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9b1db-ed33-4a5d-9364-be55f2b9b0d6 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:58.090546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9b1db-ed33-4a5d-9364-be55f2b9b0d6 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:58.090546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9b1db-ed33-4a5d-9364-be55f2b9b0d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:58.165809 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-73b9b1db-ed33-4a5d-9364-be55f2b9b0d6 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 00:59:58.167240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-73b9b1db-ed33-4a5d-9364-be55f2b9b0d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:58.186201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:58.267052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 663/1345] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 00:59:58 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 00:59:58.305051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584bb401-ac6a-43fc-94d0-a7efe583d21e None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 00:59:58.306585 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-584bb401-ac6a-43fc-94d0-a7efe583d21e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:58.421046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584bb401-ac6a-43fc-94d0-a7efe583d21e None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 00:59:58.505178 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615aeee0-248a-44ca-9095-20011814dd97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 00:59:58.505804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615aeee0-248a-44ca-9095-20011814dd97 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 00:59:58.506080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615aeee0-248a-44ca-9095-20011814dd97 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 00:59:58.506348 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615aeee0-248a-44ca-9095-20011814dd97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 00:59:58.508026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-615aeee0-248a-44ca-9095-20011814dd97 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 00:59:58.559770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 00:59:58.629644 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615aeee0-248a-44ca-9095-20011814dd97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 00:59:58.631087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615aeee0-248a-44ca-9095-20011814dd97 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 00:59:58.650782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 683/1346] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 00:59:58 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:02.735221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fa02d3-6e38-4cc8-91f0-acd231b880c1 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:02.735819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fa02d3-6e38-4cc8-91f0-acd231b880c1 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:02.735819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fa02d3-6e38-4cc8-91f0-acd231b880c1 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:02.737203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 664/1347] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:02.742825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bda125e-3b8d-40eb-a05c-8da76e045b33 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:02.743463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bda125e-3b8d-40eb-a05c-8da76e045b33 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:02.743463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bda125e-3b8d-40eb-a05c-8da76e045b33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:02.814837 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-8bda125e-3b8d-40eb-a05c-8da76e045b33 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:02.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8bda125e-3b8d-40eb-a05c-8da76e045b33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:02.917760 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:02.921301 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 684/1348] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:02.959948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fa02d3-6e38-4cc8-91f0-acd231b880c1 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:02.961229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-43fa02d3-6e38-4cc8-91f0-acd231b880c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:03.094037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fa02d3-6e38-4cc8-91f0-acd231b880c1 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:03.146719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:03.176513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4fa7d8-969f-4a0c-9541-737d552ebfc7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:03.177276 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4fa7d8-969f-4a0c-9541-737d552ebfc7 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:03.177619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4fa7d8-969f-4a0c-9541-737d552ebfc7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:03.178025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4fa7d8-969f-4a0c-9541-737d552ebfc7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:03.179836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0a4fa7d8-969f-4a0c-9541-737d552ebfc7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:03.311068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4fa7d8-969f-4a0c-9541-737d552ebfc7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:03.312546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4fa7d8-969f-4a0c-9541-737d552ebfc7 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:03.333788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 665/1349] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:08.355304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b17ee6-bd0b-4502-b838-2db43fed96fd None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:08.356234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b17ee6-bd0b-4502-b838-2db43fed96fd None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:08.356751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b17ee6-bd0b-4502-b838-2db43fed96fd None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:08.359281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 685/1350] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:00:08 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:08.369731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e48229-5418-4364-82dc-031d77594cee None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:08.370217 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e48229-5418-4364-82dc-031d77594cee None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:08.370518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e48229-5418-4364-82dc-031d77594cee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:08.481794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c7e48229-5418-4364-82dc-031d77594cee None None] MFA Rules not processed for user `1d562ccd50e4492281a9f435f809d49b`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:08.483634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c7e48229-5418-4364-82dc-031d77594cee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:08.515840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:08.673745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 666/1351] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:00:08 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:13.009634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e978ab1e-51ea-4d7e-aa7d-2b32f09bc479 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:13.010602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e978ab1e-51ea-4d7e-aa7d-2b32f09bc479 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:13.010986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e978ab1e-51ea-4d7e-aa7d-2b32f09bc479 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:13.012468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 686/1352] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:00:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:13.019766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc61c427-268f-4368-a8d5-529132be1203 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:13.020253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc61c427-268f-4368-a8d5-529132be1203 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:13.020340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc61c427-268f-4368-a8d5-529132be1203 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:13.107143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-bc61c427-268f-4368-a8d5-529132be1203 None None] MFA Rules not processed for user `1d562ccd50e4492281a9f435f809d49b`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:13.108394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-bc61c427-268f-4368-a8d5-529132be1203 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:13.184439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:13.225174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 667/1353] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:00:13 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:14.583759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b4097c-39ac-42d2-8899-5629d9fdf881 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:14.584319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b4097c-39ac-42d2-8899-5629d9fdf881 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:14.584319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b4097c-39ac-42d2-8899-5629d9fdf881 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:14.585705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 687/1354] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:00:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:14.591229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ed7e5d-2eb1-4097-bc46-3ebf04f791bd None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:14.592401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ed7e5d-2eb1-4097-bc46-3ebf04f791bd None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:14.592498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ed7e5d-2eb1-4097-bc46-3ebf04f791bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:14.676104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c6ed7e5d-2eb1-4097-bc46-3ebf04f791bd None None] MFA Rules not processed for user `1d562ccd50e4492281a9f435f809d49b`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:14.677307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c6ed7e5d-2eb1-4097-bc46-3ebf04f791bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:14.785242 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 668/1355] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:00:14 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:18.229994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d3f7d7-38db-47a1-849e-cda99d7ed223 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:18.230535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d3f7d7-38db-47a1-849e-cda99d7ed223 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:18.230895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d3f7d7-38db-47a1-849e-cda99d7ed223 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:18.234390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 688/1356] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:18 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:18.240623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92083db-f93e-47ef-9ec2-ceeb574767d4 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:18.240623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92083db-f93e-47ef-9ec2-ceeb574767d4 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:18.241427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92083db-f93e-47ef-9ec2-ceeb574767d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:18.291103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:18.332525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c92083db-f93e-47ef-9ec2-ceeb574767d4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:18.334133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c92083db-f93e-47ef-9ec2-ceeb574767d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:18.444590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 669/1357] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:18.516870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ade553a-e90c-4e55-9879-b4409f01823c None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:18.518173 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ade553a-e90c-4e55-9879-b4409f01823c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:18.519684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ade553a-e90c-4e55-9879-b4409f01823c None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:18.523964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 689/1358] 10.222.0.42 () {54 vars in 962 bytes} [Sun Jul 31 01:00:18 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:18.535070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37cf769-54a0-42c8-b38a-86f9046b7a42 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:18.535887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37cf769-54a0-42c8-b38a-86f9046b7a42 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:18.536399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37cf769-54a0-42c8-b38a-86f9046b7a42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:18.627272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c37cf769-54a0-42c8-b38a-86f9046b7a42 None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:18.628946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c37cf769-54a0-42c8-b38a-86f9046b7a42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:18.687554 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:18.791874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 670/1359] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Jul 31 01:00:18 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:18.800965 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ade553a-e90c-4e55-9879-b4409f01823c None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:18.802476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9ade553a-e90c-4e55-9879-b4409f01823c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:18.962584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ade553a-e90c-4e55-9879-b4409f01823c None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:19.038830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:19.081367 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaebf4e6-d509-4261-8c33-be37bc19c44e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:19.082013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaebf4e6-d509-4261-8c33-be37bc19c44e service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:19.082251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaebf4e6-d509-4261-8c33-be37bc19c44e service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:19.082594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaebf4e6-d509-4261-8c33-be37bc19c44e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:19.084146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-eaebf4e6-d509-4261-8c33-be37bc19c44e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:19.186154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaebf4e6-d509-4261-8c33-be37bc19c44e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:19.187777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaebf4e6-d509-4261-8c33-be37bc19c44e service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:19.207866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 690/1360] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:21.489646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81cbf79-0ee1-4a5f-9caa-1dbd6d1f6733 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:21.490196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81cbf79-0ee1-4a5f-9caa-1dbd6d1f6733 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:21.490196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81cbf79-0ee1-4a5f-9caa-1dbd6d1f6733 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:21.491752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 671/1361] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:21.496848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e743e954-1848-4341-b189-d79f74725f61 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:21.497214 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e743e954-1848-4341-b189-d79f74725f61 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:21.497401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e743e954-1848-4341-b189-d79f74725f61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:21.627407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-e743e954-1848-4341-b189-d79f74725f61 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:21.630131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e743e954-1848-4341-b189-d79f74725f61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:21.702640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:21.779183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 691/1362] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:21.828444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81cbf79-0ee1-4a5f-9caa-1dbd6d1f6733 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:21.830343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a81cbf79-0ee1-4a5f-9caa-1dbd6d1f6733 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:22.010305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81cbf79-0ee1-4a5f-9caa-1dbd6d1f6733 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:22.025154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:22.149377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f31f842-78dd-47e7-9684-e8f3ef273c78 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:22.151039 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f31f842-78dd-47e7-9684-e8f3ef273c78 service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:22.151533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f31f842-78dd-47e7-9684-e8f3ef273c78 service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:22.151968 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f31f842-78dd-47e7-9684-e8f3ef273c78 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:22.153995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6f31f842-78dd-47e7-9684-e8f3ef273c78 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:22.261067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f31f842-78dd-47e7-9684-e8f3ef273c78 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:22.262659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f31f842-78dd-47e7-9684-e8f3ef273c78 service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:22.284654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 672/1363] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:24.676406 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589065ad-5a3c-4f70-8c30-3b07ddac5aaa None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:24.676958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589065ad-5a3c-4f70-8c30-3b07ddac5aaa None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:24.677049 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589065ad-5a3c-4f70-8c30-3b07ddac5aaa None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:24.678435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 692/1364] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:24.684079 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c424c1b-4c27-4d50-8835-94ef63c93d8e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:24.684609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c424c1b-4c27-4d50-8835-94ef63c93d8e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:24.684714 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c424c1b-4c27-4d50-8835-94ef63c93d8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:24.711120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:24.778064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-9c424c1b-4c27-4d50-8835-94ef63c93d8e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:24.779497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9c424c1b-4c27-4d50-8835-94ef63c93d8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:24.903472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 673/1365] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:24.951119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259cc19e-bfe8-44d8-a00c-5da11d6c59b3 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:24.951435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259cc19e-bfe8-44d8-a00c-5da11d6c59b3 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:24.951744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259cc19e-bfe8-44d8-a00c-5da11d6c59b3 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:24.954090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 693/1366] 10.222.0.42 () {54 vars in 962 bytes} [Sun Jul 31 01:00:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:24.960075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee93c2e-56e7-4b6b-9e3a-84fc63d1646c None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:24.960365 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee93c2e-56e7-4b6b-9e3a-84fc63d1646c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:24.960644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee93c2e-56e7-4b6b-9e3a-84fc63d1646c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:25.051907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-5ee93c2e-56e7-4b6b-9e3a-84fc63d1646c None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:25.053110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5ee93c2e-56e7-4b6b-9e3a-84fc63d1646c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:25.094229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:25.192424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 674/1367] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Jul 31 01:00:24 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:25.200191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259cc19e-bfe8-44d8-a00c-5da11d6c59b3 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:25.201742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-259cc19e-bfe8-44d8-a00c-5da11d6c59b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:25.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259cc19e-bfe8-44d8-a00c-5da11d6c59b3 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:25.403714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:25.517496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5db1746-77bc-4c4b-ac3e-f56a884cd8b9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:25.518156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5db1746-77bc-4c4b-ac3e-f56a884cd8b9 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:25.518408 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5db1746-77bc-4c4b-ac3e-f56a884cd8b9 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:25.518626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5db1746-77bc-4c4b-ac3e-f56a884cd8b9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:25.520407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e5db1746-77bc-4c4b-ac3e-f56a884cd8b9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:25.660004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5db1746-77bc-4c4b-ac3e-f56a884cd8b9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:25.662723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5db1746-77bc-4c4b-ac3e-f56a884cd8b9 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:25.691866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 694/1368] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 492 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:26.536284 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e607155-5bcf-481f-9ebd-b6e91edd9646 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:26.536875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e607155-5bcf-481f-9ebd-b6e91edd9646 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:26.537215 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e607155-5bcf-481f-9ebd-b6e91edd9646 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:26.538575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 675/1369] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:26.548419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a82c480-7c25-442a-b127-d63d64a280a1 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:26.548419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a82c480-7c25-442a-b127-d63d64a280a1 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:26.548419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a82c480-7c25-442a-b127-d63d64a280a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:26.580455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:26.625201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-4a82c480-7c25-442a-b127-d63d64a280a1 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:26.626530 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4a82c480-7c25-442a-b127-d63d64a280a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:26.739211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 695/1370] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:27.101701 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c971808-74f2-48cd-9596-18b84a15d355 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:27.102135 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c971808-74f2-48cd-9596-18b84a15d355 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:27.102509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c971808-74f2-48cd-9596-18b84a15d355 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:27.104385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 676/1371] 10.222.0.42 () {54 vars in 964 bytes} [Sun Jul 31 01:00:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:27.111214 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36e2e7e-5a24-4c9b-a3d8-cbf1b7fe9137 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:27.111786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36e2e7e-5a24-4c9b-a3d8-cbf1b7fe9137 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:27.112291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36e2e7e-5a24-4c9b-a3d8-cbf1b7fe9137 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:27.218566 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f36e2e7e-5a24-4c9b-a3d8-cbf1b7fe9137 None None] MFA Rules not processed for user `9c8338bf95d34b228176b556afd93155`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:27.219820 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f36e2e7e-5a24-4c9b-a3d8-cbf1b7fe9137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:27.272049 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:27.311753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 696/1372] 10.222.0.42 () {58 vars in 1061 bytes} [Sun Jul 31 01:00:27 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:27.315675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c971808-74f2-48cd-9596-18b84a15d355 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:27.317501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-3c971808-74f2-48cd-9596-18b84a15d355 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:27.446553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c971808-74f2-48cd-9596-18b84a15d355 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:27.495143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:27.536438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b01903-82bb-42d9-be67-adbd323df7a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:27.537351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b01903-82bb-42d9-be67-adbd323df7a7 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:27.537700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b01903-82bb-42d9-be67-adbd323df7a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:27.538035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b01903-82bb-42d9-be67-adbd323df7a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:27.539801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e9b01903-82bb-42d9-be67-adbd323df7a7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:27.668360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b01903-82bb-42d9-be67-adbd323df7a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:27.670091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b01903-82bb-42d9-be67-adbd323df7a7 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:27.693840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 677/1373] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:27.852361 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7cab0-78eb-4e2a-b248-2c0ff05bd04e None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:27.852786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7cab0-78eb-4e2a-b248-2c0ff05bd04e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:27.852947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7cab0-78eb-4e2a-b248-2c0ff05bd04e None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:27.854604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 697/1374] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:27.860584 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3741991e-bc8c-448d-a67b-9be67edc2336 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:27.860904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3741991e-bc8c-448d-a67b-9be67edc2336 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:27.861164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3741991e-bc8c-448d-a67b-9be67edc2336 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:27.979596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-3741991e-bc8c-448d-a67b-9be67edc2336 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:27.981841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-3741991e-bc8c-448d-a67b-9be67edc2336 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:28.032190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:28.140375 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 678/1375] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:28.185176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a7cab0-78eb-4e2a-b248-2c0ff05bd04e None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:28.187533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c4a7cab0-78eb-4e2a-b248-2c0ff05bd04e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:28.345744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a7cab0-78eb-4e2a-b248-2c0ff05bd04e None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:28.403027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:28.477634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e32b0ef-e63c-42cc-9649-a376ff5b4506 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:28.478254 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e32b0ef-e63c-42cc-9649-a376ff5b4506 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:28.478529 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e32b0ef-e63c-42cc-9649-a376ff5b4506 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:28.478811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e32b0ef-e63c-42cc-9649-a376ff5b4506 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:28.480497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3e32b0ef-e63c-42cc-9649-a376ff5b4506 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:28.590575 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e32b0ef-e63c-42cc-9649-a376ff5b4506 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:28.595210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e32b0ef-e63c-42cc-9649-a376ff5b4506 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:28.624336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 698/1376] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:30.715151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5f5a0f-f49b-4b2e-bde1-03c1675c93c5 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:30.715735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5f5a0f-f49b-4b2e-bde1-03c1675c93c5 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:30.715735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5f5a0f-f49b-4b2e-bde1-03c1675c93c5 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:30.717101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 679/1377] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:30.722673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22eb140-dcd2-4113-a27e-140bce15bc46 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:30.723104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22eb140-dcd2-4113-a27e-140bce15bc46 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:30.723209 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22eb140-dcd2-4113-a27e-140bce15bc46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:30.796654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-c22eb140-dcd2-4113-a27e-140bce15bc46 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:30.797907 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c22eb140-dcd2-4113-a27e-140bce15bc46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:30.804635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:30.917158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 699/1378] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:30.963067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5f5a0f-f49b-4b2e-bde1-03c1675c93c5 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:30.964724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-4b5f5a0f-f49b-4b2e-bde1-03c1675c93c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:31.125698 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5f5a0f-f49b-4b2e-bde1-03c1675c93c5 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:31.129501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:31.279316 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ae48f1-2875-49c5-956e-95994bccc6e9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:31.280073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ae48f1-2875-49c5-956e-95994bccc6e9 service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:31.280356 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ae48f1-2875-49c5-956e-95994bccc6e9 service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:31.281210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ae48f1-2875-49c5-956e-95994bccc6e9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:31.282844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-04ae48f1-2875-49c5-956e-95994bccc6e9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:31.416772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ae48f1-2875-49c5-956e-95994bccc6e9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:31.418675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ae48f1-2875-49c5-956e-95994bccc6e9 service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:31.442369 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 680/1379] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:33.537722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec479ea8-47c5-41cf-8484-d17a29b8e4b8 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:33.538271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec479ea8-47c5-41cf-8484-d17a29b8e4b8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:33.538271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec479ea8-47c5-41cf-8484-d17a29b8e4b8 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:33.540059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 700/1380] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:33.548814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795be285-bd1c-4200-af5f-2c843a397347 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:33.548814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795be285-bd1c-4200-af5f-2c843a397347 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:33.548814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795be285-bd1c-4200-af5f-2c843a397347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:33.559470 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:33.624237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-795be285-bd1c-4200-af5f-2c843a397347 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:33.625454 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-795be285-bd1c-4200-af5f-2c843a397347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:33.745048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 681/1381] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:33.791892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec479ea8-47c5-41cf-8484-d17a29b8e4b8 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:33.793670 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ec479ea8-47c5-41cf-8484-d17a29b8e4b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:34.028712 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:34.049865 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec479ea8-47c5-41cf-8484-d17a29b8e4b8 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:34.245128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee542b9-7f9a-42bd-a65e-352f3c1bef82 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:34.246162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee542b9-7f9a-42bd-a65e-352f3c1bef82 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:34.246622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee542b9-7f9a-42bd-a65e-352f3c1bef82 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:34.247227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee542b9-7f9a-42bd-a65e-352f3c1bef82 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:34.250758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2ee542b9-7f9a-42bd-a65e-352f3c1bef82 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:34.408222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee542b9-7f9a-42bd-a65e-352f3c1bef82 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:34.410821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee542b9-7f9a-42bd-a65e-352f3c1bef82 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:34.429698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:34.445062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 701/1382] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 654 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:36.809073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aed281b-394c-45e5-9f9e-841f4743d8cd None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:36.809606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aed281b-394c-45e5-9f9e-841f4743d8cd None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:36.809606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aed281b-394c-45e5-9f9e-841f4743d8cd None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:36.810965 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 682/1383] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:36.817332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ea59c3-5f2b-4375-9d70-d0a53f4cccb1 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:36.817953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ea59c3-5f2b-4375-9d70-d0a53f4cccb1 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:36.817953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ea59c3-5f2b-4375-9d70-d0a53f4cccb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:36.886805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-38ea59c3-5f2b-4375-9d70-d0a53f4cccb1 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:36.888067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-38ea59c3-5f2b-4375-9d70-d0a53f4cccb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:36.991018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 702/1384] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:37.033129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aed281b-394c-45e5-9f9e-841f4743d8cd None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:37.034635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-3aed281b-394c-45e5-9f9e-841f4743d8cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:37.126098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:37.203239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aed281b-394c-45e5-9f9e-841f4743d8cd None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:37.322012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192616b3-fa83-47a0-b874-34e9601fc86d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:37.322627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192616b3-fa83-47a0-b874-34e9601fc86d service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:37.322906 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192616b3-fa83-47a0-b874-34e9601fc86d service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:37.323178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192616b3-fa83-47a0-b874-34e9601fc86d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:37.324817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-192616b3-fa83-47a0-b874-34e9601fc86d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:37.377442 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:37.421868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192616b3-fa83-47a0-b874-34e9601fc86d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:37.423519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192616b3-fa83-47a0-b874-34e9601fc86d service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:37.442707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 683/1385] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:39.489343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528485a7-3b34-4b10-ad3a-e02c6912ae21 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:39.489873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528485a7-3b34-4b10-ad3a-e02c6912ae21 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:39.490260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528485a7-3b34-4b10-ad3a-e02c6912ae21 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:39.492893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 703/1386] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:39 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:39.498443 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1b5f0f-7e40-476c-afce-62f3753eb953 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:39.498904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1b5f0f-7e40-476c-afce-62f3753eb953 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:39.498999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1b5f0f-7e40-476c-afce-62f3753eb953 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:39.594912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-5a1b5f0f-7e40-476c-afce-62f3753eb953 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:39.597308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5a1b5f0f-7e40-476c-afce-62f3753eb953 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:39.722231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 684/1387] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:39.770477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528485a7-3b34-4b10-ad3a-e02c6912ae21 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:39.772080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-528485a7-3b34-4b10-ad3a-e02c6912ae21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:39.865617 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:39.939561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528485a7-3b34-4b10-ad3a-e02c6912ae21 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:40.051949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b797dc-8a2e-4662-8662-d5dbf36ef69c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:40.052487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b797dc-8a2e-4662-8662-d5dbf36ef69c service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:40.052764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b797dc-8a2e-4662-8662-d5dbf36ef69c service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:40.053033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b797dc-8a2e-4662-8662-d5dbf36ef69c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:40.054665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-27b797dc-8a2e-4662-8662-d5dbf36ef69c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:40.092730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:40.147873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b797dc-8a2e-4662-8662-d5dbf36ef69c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:40.149493 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b797dc-8a2e-4662-8662-d5dbf36ef69c service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:40.171420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 704/1388] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:42.248394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e09db-8b7e-41f0-8083-710ad5875513 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:42.249024 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e09db-8b7e-41f0-8083-710ad5875513 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:42.249024 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e09db-8b7e-41f0-8083-710ad5875513 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:42.251202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 685/1389] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:42.256525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03583c21-d4a2-410e-8c51-437086386973 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:42.256987 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03583c21-d4a2-410e-8c51-437086386973 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:42.257100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03583c21-d4a2-410e-8c51-437086386973 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:42.323419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-03583c21-d4a2-410e-8c51-437086386973 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:42.324627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-03583c21-d4a2-410e-8c51-437086386973 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:42.424657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 705/1390] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:42.470776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e09db-8b7e-41f0-8083-710ad5875513 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:42.472335 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2f5e09db-8b7e-41f0-8083-710ad5875513 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:42.504774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:42.628179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e09db-8b7e-41f0-8083-710ad5875513 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:42.728944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:42.742750 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ad2b78-4ba5-4a9f-82e4-63f310f60a3b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:42.743384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ad2b78-4ba5-4a9f-82e4-63f310f60a3b service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:42.743670 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ad2b78-4ba5-4a9f-82e4-63f310f60a3b service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:42.743945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ad2b78-4ba5-4a9f-82e4-63f310f60a3b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:42.745535 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d1ad2b78-4ba5-4a9f-82e4-63f310f60a3b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:42.835828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ad2b78-4ba5-4a9f-82e4-63f310f60a3b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:42.837426 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ad2b78-4ba5-4a9f-82e4-63f310f60a3b service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:42.856247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 686/1391] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:43.268765 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6d5716-8e11-409e-8219-cfb610a38b70 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:43.269333 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6d5716-8e11-409e-8219-cfb610a38b70 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:43.269454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6d5716-8e11-409e-8219-cfb610a38b70 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:43.270816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 706/1392] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:43.276102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b686a60-bf07-4c7e-b1b8-98089109e05b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:43.276568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b686a60-bf07-4c7e-b1b8-98089109e05b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:43.277061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b686a60-bf07-4c7e-b1b8-98089109e05b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:43.345587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-7b686a60-bf07-4c7e-b1b8-98089109e05b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:43.346803 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7b686a60-bf07-4c7e-b1b8-98089109e05b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:43.406162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:43.446729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 687/1393] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:43.505396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e08c65e-d05f-4fd2-821d-c9118679cc10 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:43.505676 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e08c65e-d05f-4fd2-821d-c9118679cc10 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:43.505951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e08c65e-d05f-4fd2-821d-c9118679cc10 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:43.507451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 707/1394] 10.222.0.42 () {54 vars in 966 bytes} [Sun Jul 31 01:00:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:43.512335 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9f67b3-68c9-452b-9f82-a90b53669906 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:43.512617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9f67b3-68c9-452b-9f82-a90b53669906 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:43.512888 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9f67b3-68c9-452b-9f82-a90b53669906 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:43.593525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-1a9f67b3-68c9-452b-9f82-a90b53669906 None None] MFA Rules not processed for user `4160039561434e51a87c6c7a019f2dff`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:43.594729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1a9f67b3-68c9-452b-9f82-a90b53669906 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:43.672954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 688/1395] 10.222.0.42 () {58 vars in 1063 bytes} [Sun Jul 31 01:00:43 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:43.680456 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e08c65e-d05f-4fd2-821d-c9118679cc10 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:43.682018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0e08c65e-d05f-4fd2-821d-c9118679cc10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:43.696098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:43.788715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e08c65e-d05f-4fd2-821d-c9118679cc10 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:43.872051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81429d61-e4a8-4629-9728-1191775179b6 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4160039561434e51a87c6c7a019f2dff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:43.872600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81429d61-e4a8-4629-9728-1191775179b6 service glance] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:43.872901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81429d61-e4a8-4629-9728-1191775179b6 service glance] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:43.873165 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81429d61-e4a8-4629-9728-1191775179b6 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:43.874792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-81429d61-e4a8-4629-9728-1191775179b6 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:43.947308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:44.003134 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81429d61-e4a8-4629-9728-1191775179b6 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:44.004615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81429d61-e4a8-4629-9728-1191775179b6 service glance] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:44.025221 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 708/1396] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:00:44.768808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d07d44-da96-4e66-8639-99506b5ded7a None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:44.768808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d07d44-da96-4e66-8639-99506b5ded7a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:44.768808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d07d44-da96-4e66-8639-99506b5ded7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:44.847138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a7d07d44-da96-4e66-8639-99506b5ded7a None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:44.848379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a7d07d44-da96-4e66-8639-99506b5ded7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:44.856551 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:44.921036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f9786c-070e-4c8b-8407-b41ecb488eda None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:44.921606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f9786c-070e-4c8b-8407-b41ecb488eda None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:44.921721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f9786c-070e-4c8b-8407-b41ecb488eda None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:44.923877 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 709/1397] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 01:00:44.929139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5480954-8ea7-4e29-8bdd-facca67a5322 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:44.929416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5480954-8ea7-4e29-8bdd-facca67a5322 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:44.929686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5480954-8ea7-4e29-8bdd-facca67a5322 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:44.935687 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 689/1398] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:00:44 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:44.983296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f1cf6b-d44c-4494-847b-15f6b8ada86e None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:44.983639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f1cf6b-d44c-4494-847b-15f6b8ada86e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:44.983914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f1cf6b-d44c-4494-847b-15f6b8ada86e None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:44.985890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 690/1399] 10.222.0.42 () {54 vars in 943 bytes} [Sun Jul 31 01:00:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 01:00:44.992313 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c85e06-3385-4384-93b9-d5e8e67ddf33 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:44.992596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c85e06-3385-4384-93b9-d5e8e67ddf33 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:44.992876 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c85e06-3385-4384-93b9-d5e8e67ddf33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:45.001590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f5480954-8ea7-4e29-8bdd-facca67a5322 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:45.002941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f5480954-8ea7-4e29-8bdd-facca67a5322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:45.070692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-f0c85e06-3385-4384-93b9-d5e8e67ddf33 None None] MFA Rules not processed for user `c11f1dd5a6b042c9b864a5861d0f3569`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:45.071845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f0c85e06-3385-4384-93b9-d5e8e67ddf33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:45.103823 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 710/1400] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:00:45.150581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5480954-8ea7-4e29-8bdd-facca67a5322 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:45.151812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f5480954-8ea7-4e29-8bdd-facca67a5322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:45.164804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:45.181113 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:45.190951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 691/1401] 10.222.0.42 () {58 vars in 1040 bytes} [Sun Jul 31 01:00:44 2022] POST /identity/v3/auth/tokens => generated 3294 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:00:45.200530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c85e06-3385-4384-93b9-d5e8e67ddf33 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:45.210615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c85e06-3385-4384-93b9-d5e8e67ddf33 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:45.271739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5480954-8ea7-4e29-8bdd-facca67a5322 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:45.317543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7a0af2-9b92-4f99-bbc6-6e11aaa079b9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:45.318173 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7a0af2-9b92-4f99-bbc6-6e11aaa079b9 service swift] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:45.318449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7a0af2-9b92-4f99-bbc6-6e11aaa079b9 service swift] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:45.318716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7a0af2-9b92-4f99-bbc6-6e11aaa079b9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:45.320448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1a7a0af2-9b92-4f99-bbc6-6e11aaa079b9 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:45.388408 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d5788a-44d1-457f-aef0-8c8432738b5b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:45.389017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d5788a-44d1-457f-aef0-8c8432738b5b service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:45.389297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d5788a-44d1-457f-aef0-8c8432738b5b service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:45.389560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d5788a-44d1-457f-aef0-8c8432738b5b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:45.391759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-87d5788a-44d1-457f-aef0-8c8432738b5b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:45.404554 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7a0af2-9b92-4f99-bbc6-6e11aaa079b9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:45.406137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7a0af2-9b92-4f99-bbc6-6e11aaa079b9 service swift] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:45.410730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:45.414854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 692/1402] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:00:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 31 01:00:45.486318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d5788a-44d1-457f-aef0-8c8432738b5b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:45.488250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d5788a-44d1-457f-aef0-8c8432738b5b service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:45.508280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 711/1403] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:00:47.562617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9523717-6368-42db-adef-33050262a13b None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:47.563729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9523717-6368-42db-adef-33050262a13b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:47.563729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9523717-6368-42db-adef-33050262a13b None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:47.565250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 693/1404] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:47.570373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1423a636-bdf1-49c0-bb4c-defc6c81e20d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:47.570893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1423a636-bdf1-49c0-bb4c-defc6c81e20d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:47.570893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1423a636-bdf1-49c0-bb4c-defc6c81e20d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:47.648480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-1423a636-bdf1-49c0-bb4c-defc6c81e20d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:47.650265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-1423a636-bdf1-49c0-bb4c-defc6c81e20d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:47.776913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:47.785776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 712/1405] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:47.831615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9523717-6368-42db-adef-33050262a13b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:47.833398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e9523717-6368-42db-adef-33050262a13b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:47.844465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:48.008286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9523717-6368-42db-adef-33050262a13b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:48.098908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:48.143466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47856704-845a-4b56-b13a-0f10b9edfc05 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:48.145568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47856704-845a-4b56-b13a-0f10b9edfc05 service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:48.146091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47856704-845a-4b56-b13a-0f10b9edfc05 service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:48.146410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47856704-845a-4b56-b13a-0f10b9edfc05 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:48.148682 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-47856704-845a-4b56-b13a-0f10b9edfc05 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:48.259814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47856704-845a-4b56-b13a-0f10b9edfc05 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:48.261863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47856704-845a-4b56-b13a-0f10b9edfc05 service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:48.284549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 694/1406] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:00:50.349670 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621c3692-36ee-4e5d-b433-13d6588415ba None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:50.350235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621c3692-36ee-4e5d-b433-13d6588415ba None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:50.350235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621c3692-36ee-4e5d-b433-13d6588415ba None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:50.351582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 713/1407] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:00:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:00:50.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7a03e0-595f-48e7-a3c7-96235faab86f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:50.357098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7a03e0-595f-48e7-a3c7-96235faab86f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:50.357192 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7a03e0-595f-48e7-a3c7-96235faab86f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:50.425955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6c7a03e0-595f-48e7-a3c7-96235faab86f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:00:50.427627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6c7a03e0-595f-48e7-a3c7-96235faab86f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:50.483710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:50.573810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 695/1408] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:00:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:00:50.621496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621c3692-36ee-4e5d-b433-13d6588415ba None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:00:50.623029 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-621c3692-36ee-4e5d-b433-13d6588415ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:50.793049 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621c3692-36ee-4e5d-b433-13d6588415ba None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:00:50.885270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:00:50.946696 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13184f2-f135-4b31-9d2c-b2f9a7e05d82 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:00:50.947309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13184f2-f135-4b31-9d2c-b2f9a7e05d82 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:00:50.947630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13184f2-f135-4b31-9d2c-b2f9a7e05d82 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:00:50.947900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13184f2-f135-4b31-9d2c-b2f9a7e05d82 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:00:50.949505 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a13184f2-f135-4b31-9d2c-b2f9a7e05d82 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:00:51.057730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13184f2-f135-4b31-9d2c-b2f9a7e05d82 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:00:51.059242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13184f2-f135-4b31-9d2c-b2f9a7e05d82 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:00:51.080652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 714/1409] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:00:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:01:00.571304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=23 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:01.078148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=64 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:07.549041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71d0b56-1a36-4fbc-b58f-734e46b48164 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:07.549571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71d0b56-1a36-4fbc-b58f-734e46b48164 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:07.549571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71d0b56-1a36-4fbc-b58f-734e46b48164 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:07.550937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 696/1410] 10.222.0.42 () {54 vars in 918 bytes} [Sun Jul 31 01:01:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:07.556191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4563e656-f4f3-493d-b090-89ab9f4f281c None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:07.556589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4563e656-f4f3-493d-b090-89ab9f4f281c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:07.556766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4563e656-f4f3-493d-b090-89ab9f4f281c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:07.632841 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-4563e656-f4f3-493d-b090-89ab9f4f281c None None] MFA Rules not processed for user `1d562ccd50e4492281a9f435f809d49b`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:07.633953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4563e656-f4f3-493d-b090-89ab9f4f281c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:07.738530 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 715/1411] 10.222.0.42 () {58 vars in 1015 bytes} [Sun Jul 31 01:01:07 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:07.754233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ce30ae-a5e3-4740-bca1-027b123fd93e None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:07.754522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ce30ae-a5e3-4740-bca1-027b123fd93e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:07.754801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ce30ae-a5e3-4740-bca1-027b123fd93e None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:07.756187 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 697/1412] 10.222.0.42 () {54 vars in 918 bytes} [Sun Jul 31 01:01:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:07.760807 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07af9633-b923-4e1c-8d27-c68ce1562c69 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:07.761094 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07af9633-b923-4e1c-8d27-c68ce1562c69 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:07.761371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07af9633-b923-4e1c-8d27-c68ce1562c69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:07.833407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:07.836387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-07af9633-b923-4e1c-8d27-c68ce1562c69 None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:07.838171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-07af9633-b923-4e1c-8d27-c68ce1562c69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:07.954221 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 716/1413] 10.222.0.42 () {58 vars in 1015 bytes} [Sun Jul 31 01:01:07 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:10.430581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a300765-fa9b-4760-9156-57eade2aeca1 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:10.431159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a300765-fa9b-4760-9156-57eade2aeca1 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:10.431159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a300765-fa9b-4760-9156-57eade2aeca1 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:10.432562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 698/1414] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:01:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:10.437855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba0af99-4041-4393-ab31-0f89a728ba8b None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:10.438290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba0af99-4041-4393-ab31-0f89a728ba8b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:10.438397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba0af99-4041-4393-ab31-0f89a728ba8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:10.625691 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-fba0af99-4041-4393-ab31-0f89a728ba8b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:10.626923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-fba0af99-4041-4393-ab31-0f89a728ba8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:10.682713 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:10.725140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 717/1415] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:01:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:10.767830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a300765-fa9b-4760-9156-57eade2aeca1 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:10.769529 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1a300765-fa9b-4760-9156-57eade2aeca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:10.911982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a300765-fa9b-4760-9156-57eade2aeca1 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:10.983292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:11.023076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3763254-24ae-4822-8966-0fb0e85a8254 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:11.023707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3763254-24ae-4822-8966-0fb0e85a8254 service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:11.024101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3763254-24ae-4822-8966-0fb0e85a8254 service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:11.024368 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3763254-24ae-4822-8966-0fb0e85a8254 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:11.025978 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e3763254-24ae-4822-8966-0fb0e85a8254 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:11.115247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3763254-24ae-4822-8966-0fb0e85a8254 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:11.116866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3763254-24ae-4822-8966-0fb0e85a8254 service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:11.135969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 699/1416] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:12.043983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6698ef-24dc-4fea-9e73-c3db19dcf010 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:12.044547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6698ef-24dc-4fea-9e73-c3db19dcf010 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:12.044547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6698ef-24dc-4fea-9e73-c3db19dcf010 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:12.045826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 718/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 31 01:01:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:12.051719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca28231-5c5c-4d74-a5e3-137f05ff5dc0 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:12.052176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca28231-5c5c-4d74-a5e3-137f05ff5dc0 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:12.052280 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca28231-5c5c-4d74-a5e3-137f05ff5dc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:12.126722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-fca28231-5c5c-4d74-a5e3-137f05ff5dc0 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:12.127941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fca28231-5c5c-4d74-a5e3-137f05ff5dc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:12.154496 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:12.224785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 700/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 31 01:01:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:12.377682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6698ef-24dc-4fea-9e73-c3db19dcf010 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:12.379095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2e6698ef-24dc-4fea-9e73-c3db19dcf010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:12.523013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6698ef-24dc-4fea-9e73-c3db19dcf010 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:12.565039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:12.637295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b12f818-270b-4044-bbd0-054556a42d1b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:12.637937 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b12f818-270b-4044-bbd0-054556a42d1b service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:12.638213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b12f818-270b-4044-bbd0-054556a42d1b service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:12.638477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b12f818-270b-4044-bbd0-054556a42d1b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:12.640099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6b12f818-270b-4044-bbd0-054556a42d1b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:12.728634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b12f818-270b-4044-bbd0-054556a42d1b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:12.730203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b12f818-270b-4044-bbd0-054556a42d1b service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:12.747506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 719/1419] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:13.635587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cc7a9a-38a8-4f23-89b6-57ddebf26e5c None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:13.636103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cc7a9a-38a8-4f23-89b6-57ddebf26e5c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:13.636103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cc7a9a-38a8-4f23-89b6-57ddebf26e5c None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:13.637425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 701/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 31 01:01:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:13.642952 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fef10d-8758-41fb-881d-491671a0510e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:13.643416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fef10d-8758-41fb-881d-491671a0510e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:13.643525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fef10d-8758-41fb-881d-491671a0510e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:13.697178 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:13.719931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-12fef10d-8758-41fb-881d-491671a0510e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:13.721285 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-12fef10d-8758-41fb-881d-491671a0510e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:13.817409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 720/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 31 01:01:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:13.971212 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cc7a9a-38a8-4f23-89b6-57ddebf26e5c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:13.972898 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-32cc7a9a-38a8-4f23-89b6-57ddebf26e5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:14.118791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cc7a9a-38a8-4f23-89b6-57ddebf26e5c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:14.130694 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:14.231795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f7b204-76ec-4779-b69a-b535820ccb59 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:14.232312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f7b204-76ec-4779-b69a-b535820ccb59 service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:14.232586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f7b204-76ec-4779-b69a-b535820ccb59 service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:14.232852 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f7b204-76ec-4779-b69a-b535820ccb59 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:14.234451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b1f7b204-76ec-4779-b69a-b535820ccb59 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:14.325512 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f7b204-76ec-4779-b69a-b535820ccb59 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:14.327117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f7b204-76ec-4779-b69a-b535820ccb59 service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:14.344717 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 702/1422] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:15.213276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27866fc-dac6-4ced-a1b9-4c8804626397 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:15.213864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27866fc-dac6-4ced-a1b9-4c8804626397 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:15.213864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27866fc-dac6-4ced-a1b9-4c8804626397 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:15.215180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 721/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 31 01:01:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:15.220108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b899fd4d-f0d5-4fe5-8e48-a07d28dcd99f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:15.220512 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b899fd4d-f0d5-4fe5-8e48-a07d28dcd99f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:15.220696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b899fd4d-f0d5-4fe5-8e48-a07d28dcd99f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:15.240870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:15.296820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b899fd4d-f0d5-4fe5-8e48-a07d28dcd99f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:15.298025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b899fd4d-f0d5-4fe5-8e48-a07d28dcd99f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:15.393485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 703/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 31 01:01:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:15.543451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27866fc-dac6-4ced-a1b9-4c8804626397 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:15.544892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c27866fc-dac6-4ced-a1b9-4c8804626397 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:15.654514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:15.691079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27866fc-dac6-4ced-a1b9-4c8804626397 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:15.802402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167abdf4-302e-4185-a968-687cff4a5ec2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:15.802923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167abdf4-302e-4185-a968-687cff4a5ec2 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:15.803199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167abdf4-302e-4185-a968-687cff4a5ec2 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:15.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167abdf4-302e-4185-a968-687cff4a5ec2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:15.805111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-167abdf4-302e-4185-a968-687cff4a5ec2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:15.880654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:15.894516 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167abdf4-302e-4185-a968-687cff4a5ec2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:15.896131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167abdf4-302e-4185-a968-687cff4a5ec2 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:15.914417 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 722/1425] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:16.783868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844b6b9a-ddb9-4b62-b5be-0c598643e0e4 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:16.784429 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844b6b9a-ddb9-4b62-b5be-0c598643e0e4 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:16.784429 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844b6b9a-ddb9-4b62-b5be-0c598643e0e4 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:16.785790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 704/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 31 01:01:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:16.790591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1636a1d1-a5b6-468e-b8a4-97876362146d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:16.790979 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1636a1d1-a5b6-468e-b8a4-97876362146d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:16.791146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1636a1d1-a5b6-468e-b8a4-97876362146d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:16.865788 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-1636a1d1-a5b6-468e-b8a4-97876362146d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:16.866912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-1636a1d1-a5b6-468e-b8a4-97876362146d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:16.960895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 723/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 31 01:01:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:17.112872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844b6b9a-ddb9-4b62-b5be-0c598643e0e4 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:17.114393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-844b6b9a-ddb9-4b62-b5be-0c598643e0e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:17.198319 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:17.260165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844b6b9a-ddb9-4b62-b5be-0c598643e0e4 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:17.377203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f35d2d-9368-4c6f-8134-c03a3149128d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:17.377817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f35d2d-9368-4c6f-8134-c03a3149128d service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:17.378093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f35d2d-9368-4c6f-8134-c03a3149128d service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:17.378360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f35d2d-9368-4c6f-8134-c03a3149128d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:17.379984 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f3f35d2d-9368-4c6f-8134-c03a3149128d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:17.429357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:17.472190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f35d2d-9368-4c6f-8134-c03a3149128d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:17.473734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f35d2d-9368-4c6f-8134-c03a3149128d service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:17.491852 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 705/1428] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:18.302251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e82f69f-415d-4746-8b98-a6f3b3e987f3 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:18.302826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e82f69f-415d-4746-8b98-a6f3b3e987f3 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:18.302826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e82f69f-415d-4746-8b98-a6f3b3e987f3 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:18.304237 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 724/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 31 01:01:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:18.309095 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32744c0f-2934-49c6-8ddb-f63204f02bae None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:18.309487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32744c0f-2934-49c6-8ddb-f63204f02bae None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:18.309664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32744c0f-2934-49c6-8ddb-f63204f02bae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:18.384631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-32744c0f-2934-49c6-8ddb-f63204f02bae None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:18.385855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-32744c0f-2934-49c6-8ddb-f63204f02bae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:18.481032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 706/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 31 01:01:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:18.638382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e82f69f-415d-4746-8b98-a6f3b3e987f3 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:18.639868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2e82f69f-415d-4746-8b98-a6f3b3e987f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:18.692913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:18.792856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e82f69f-415d-4746-8b98-a6f3b3e987f3 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:18.908839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707f2b72-fb7a-43f7-a096-587bf4725556 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:18.909451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707f2b72-fb7a-43f7-a096-587bf4725556 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:18.909727 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707f2b72-fb7a-43f7-a096-587bf4725556 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:18.910002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707f2b72-fb7a-43f7-a096-587bf4725556 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:18.911636 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-707f2b72-fb7a-43f7-a096-587bf4725556 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:18.918628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:19.006328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707f2b72-fb7a-43f7-a096-587bf4725556 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:19.007970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707f2b72-fb7a-43f7-a096-587bf4725556 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:19.026360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 725/1431] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:19.843279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d64ee5f-dd37-4bdc-9ae6-5bf04fc52322 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:19.843878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d64ee5f-dd37-4bdc-9ae6-5bf04fc52322 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:19.843976 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d64ee5f-dd37-4bdc-9ae6-5bf04fc52322 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:19.845279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 707/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 31 01:01:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:19.850301 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f51fe1-62e3-4633-8f7a-fca4a90353bc None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:19.850733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f51fe1-62e3-4633-8f7a-fca4a90353bc None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:19.850847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f51fe1-62e3-4633-8f7a-fca4a90353bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:19.926711 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-06f51fe1-62e3-4633-8f7a-fca4a90353bc None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:19.927959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-06f51fe1-62e3-4633-8f7a-fca4a90353bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:19.993966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:20.023295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 726/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 31 01:01:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:20.178168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d64ee5f-dd37-4bdc-9ae6-5bf04fc52322 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:20.180153 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2d64ee5f-dd37-4bdc-9ae6-5bf04fc52322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:20.197024 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:20.323376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d64ee5f-dd37-4bdc-9ae6-5bf04fc52322 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:20.409193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:20.435666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db03abd1-c1b2-45a2-935d-f43ec1a8eb9a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:20.436275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db03abd1-c1b2-45a2-935d-f43ec1a8eb9a service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:20.436552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db03abd1-c1b2-45a2-935d-f43ec1a8eb9a service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:20.436835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db03abd1-c1b2-45a2-935d-f43ec1a8eb9a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:20.438421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-db03abd1-c1b2-45a2-935d-f43ec1a8eb9a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:20.531304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db03abd1-c1b2-45a2-935d-f43ec1a8eb9a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:20.533970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db03abd1-c1b2-45a2-935d-f43ec1a8eb9a service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:20.553198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 708/1434] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:21.357791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47074fa8-4c18-4a54-bff6-e91e43d8801e None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:21.358369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47074fa8-4c18-4a54-bff6-e91e43d8801e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:21.358369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47074fa8-4c18-4a54-bff6-e91e43d8801e None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:21.359742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 727/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 31 01:01:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:21.365332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820b3c94-9b37-4e50-999c-f115b6561eb4 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:21.365762 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820b3c94-9b37-4e50-999c-f115b6561eb4 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:21.365868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820b3c94-9b37-4e50-999c-f115b6561eb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:21.443145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-820b3c94-9b37-4e50-999c-f115b6561eb4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:21.444369 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-820b3c94-9b37-4e50-999c-f115b6561eb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:21.483952 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:21.540424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 709/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 31 01:01:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:21.690400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47074fa8-4c18-4a54-bff6-e91e43d8801e None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:21.692213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-47074fa8-4c18-4a54-bff6-e91e43d8801e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:21.843380 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47074fa8-4c18-4a54-bff6-e91e43d8801e None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:21.898008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:21.958193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4d0d1f-ee1f-482a-886f-aeef6a93717c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:21.958727 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4d0d1f-ee1f-482a-886f-aeef6a93717c service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:21.959007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4d0d1f-ee1f-482a-886f-aeef6a93717c service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:21.959280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4d0d1f-ee1f-482a-886f-aeef6a93717c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:21.960922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-de4d0d1f-ee1f-482a-886f-aeef6a93717c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:22.056350 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4d0d1f-ee1f-482a-886f-aeef6a93717c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:22.057968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4d0d1f-ee1f-482a-886f-aeef6a93717c service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:22.075797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 728/1437] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:22.890154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5d7409-b24a-42d1-ad73-f9d6e55108e2 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:22.890668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5d7409-b24a-42d1-ad73-f9d6e55108e2 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:22.890668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5d7409-b24a-42d1-ad73-f9d6e55108e2 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:22.892059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 710/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 31 01:01:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:22.897003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deafe0b9-2666-44da-b08b-f843c8c1320f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:22.897438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deafe0b9-2666-44da-b08b-f843c8c1320f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:22.897579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deafe0b9-2666-44da-b08b-f843c8c1320f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:22.972183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-deafe0b9-2666-44da-b08b-f843c8c1320f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:22.973381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-deafe0b9-2666-44da-b08b-f843c8c1320f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:22.975409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:23.077187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 729/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 31 01:01:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:23.229592 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5d7409-b24a-42d1-ad73-f9d6e55108e2 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:23.231092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-cb5d7409-b24a-42d1-ad73-f9d6e55108e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:23.375497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5d7409-b24a-42d1-ad73-f9d6e55108e2 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:23.401860 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:23.488015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32e26bc-465c-4b91-84d8-44fb9b546fb4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:23.488597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e26bc-465c-4b91-84d8-44fb9b546fb4 service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:23.488873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e26bc-465c-4b91-84d8-44fb9b546fb4 service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:23.489142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e26bc-465c-4b91-84d8-44fb9b546fb4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:23.490745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e32e26bc-465c-4b91-84d8-44fb9b546fb4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:23.582457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32e26bc-465c-4b91-84d8-44fb9b546fb4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:23.584119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32e26bc-465c-4b91-84d8-44fb9b546fb4 service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:23.602563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 711/1440] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:25.528734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e47d110-3b0d-4897-a96b-bc1506fe538b None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:25.529264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e47d110-3b0d-4897-a96b-bc1506fe538b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:25.529264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e47d110-3b0d-4897-a96b-bc1506fe538b None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:25.530634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 730/1441] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:01:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:25.536293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce0977e-9cc1-49ac-bc31-2d59fedf2c03 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:25.536749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce0977e-9cc1-49ac-bc31-2d59fedf2c03 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:25.536864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce0977e-9cc1-49ac-bc31-2d59fedf2c03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:25.582691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:25.607732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-cce0977e-9cc1-49ac-bc31-2d59fedf2c03 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:25.608958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-cce0977e-9cc1-49ac-bc31-2d59fedf2c03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:25.708422 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 712/1442] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:01:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:25.746982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e47d110-3b0d-4897-a96b-bc1506fe538b None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:25.748562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5e47d110-3b0d-4897-a96b-bc1506fe538b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:25.851018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e47d110-3b0d-4897-a96b-bc1506fe538b None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:25.883711 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:25.926783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc20c420-fd3b-4ae5-be67-4c4c9d77fd1d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:25.927423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc20c420-fd3b-4ae5-be67-4c4c9d77fd1d service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:25.927703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc20c420-fd3b-4ae5-be67-4c4c9d77fd1d service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:25.927970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc20c420-fd3b-4ae5-be67-4c4c9d77fd1d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:25.929579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-bc20c420-fd3b-4ae5-be67-4c4c9d77fd1d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:26.036818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc20c420-fd3b-4ae5-be67-4c4c9d77fd1d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:26.038244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc20c420-fd3b-4ae5-be67-4c4c9d77fd1d service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:26.056757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 731/1443] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:28.380221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b31711-b586-43d1-8ed4-3f6cb89ee750 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:28.380783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b31711-b586-43d1-8ed4-3f6cb89ee750 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:28.380783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b31711-b586-43d1-8ed4-3f6cb89ee750 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:28.382111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 713/1444] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:01:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:28.387313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471fd2dc-9fc0-434c-af38-c871801f7443 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:28.387783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471fd2dc-9fc0-434c-af38-c871801f7443 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:28.387880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471fd2dc-9fc0-434c-af38-c871801f7443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:28.452709 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-471fd2dc-9fc0-434c-af38-c871801f7443 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:28.453922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-471fd2dc-9fc0-434c-af38-c871801f7443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:28.466462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:28.551394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 732/1445] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:01:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:28.589277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b31711-b586-43d1-8ed4-3f6cb89ee750 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:28.590802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d5b31711-b586-43d1-8ed4-3f6cb89ee750 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:28.694556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b31711-b586-43d1-8ed4-3f6cb89ee750 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:28.707365 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:28.771028 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42e747-540e-41f7-9c0a-2c341b6077e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:28.771677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42e747-540e-41f7-9c0a-2c341b6077e8 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:28.771957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42e747-540e-41f7-9c0a-2c341b6077e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:28.772230 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42e747-540e-41f7-9c0a-2c341b6077e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:28.773845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8a42e747-540e-41f7-9c0a-2c341b6077e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:28.881093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a42e747-540e-41f7-9c0a-2c341b6077e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:28.882530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a42e747-540e-41f7-9c0a-2c341b6077e8 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:28.900924 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 714/1446] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:31.307769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f5da30-85eb-4097-8be6-7f91a2c0e815 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:31.308390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f5da30-85eb-4097-8be6-7f91a2c0e815 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:31.308390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f5da30-85eb-4097-8be6-7f91a2c0e815 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:31.309659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 733/1447] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:01:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:31.314993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c130b4-e56f-44e8-8ba3-4d1791c17b0e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:31.316007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c130b4-e56f-44e8-8ba3-4d1791c17b0e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:31.316127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c130b4-e56f-44e8-8ba3-4d1791c17b0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:31.365265 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:31.385999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d3c130b4-e56f-44e8-8ba3-4d1791c17b0e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:31.387235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d3c130b4-e56f-44e8-8ba3-4d1791c17b0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:31.614418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 715/1448] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:01:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 300 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:01:31.654264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f5da30-85eb-4097-8be6-7f91a2c0e815 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:31.655833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-98f5da30-85eb-4097-8be6-7f91a2c0e815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:31.761500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f5da30-85eb-4097-8be6-7f91a2c0e815 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:31.800791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:31.834349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a077f3e5-5c35-4e30-ba34-c14b07a3fc27 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:31.834969 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a077f3e5-5c35-4e30-ba34-c14b07a3fc27 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:31.835243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a077f3e5-5c35-4e30-ba34-c14b07a3fc27 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:31.835541 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a077f3e5-5c35-4e30-ba34-c14b07a3fc27 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:31.837145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a077f3e5-5c35-4e30-ba34-c14b07a3fc27 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:31.944930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a077f3e5-5c35-4e30-ba34-c14b07a3fc27 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:31.946361 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a077f3e5-5c35-4e30-ba34-c14b07a3fc27 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:31.964176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 734/1449] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:34.100809 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe91548-d4aa-4854-b4d6-c5367ea3a75d None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:34.101383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe91548-d4aa-4854-b4d6-c5367ea3a75d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:34.101383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe91548-d4aa-4854-b4d6-c5367ea3a75d None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:34.102754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 716/1450] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:01:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:34.108177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96eb00e4-ddd5-40f9-8672-ada74c5edc3a None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:34.108641 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96eb00e4-ddd5-40f9-8672-ada74c5edc3a None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:34.108825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96eb00e4-ddd5-40f9-8672-ada74c5edc3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:34.178183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-96eb00e4-ddd5-40f9-8672-ada74c5edc3a None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:34.179428 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-96eb00e4-ddd5-40f9-8672-ada74c5edc3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:34.199642 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:34.276303 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 735/1451] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:01:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:34.316393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe91548-d4aa-4854-b4d6-c5367ea3a75d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:34.317878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9fe91548-d4aa-4854-b4d6-c5367ea3a75d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:34.419105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe91548-d4aa-4854-b4d6-c5367ea3a75d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:34.445854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:34.499164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72cddbe-6792-41b9-ad5f-9d69e23e7edc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:34.499881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72cddbe-6792-41b9-ad5f-9d69e23e7edc service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:34.500159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72cddbe-6792-41b9-ad5f-9d69e23e7edc service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:34.500435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72cddbe-6792-41b9-ad5f-9d69e23e7edc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:34.502035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a72cddbe-6792-41b9-ad5f-9d69e23e7edc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:34.635274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72cddbe-6792-41b9-ad5f-9d69e23e7edc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:34.636765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72cddbe-6792-41b9-ad5f-9d69e23e7edc service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:34.666702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 717/1452] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:36.857759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f973d-d1ea-41ae-978a-9384e43cc574 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:36.858290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f973d-d1ea-41ae-978a-9384e43cc574 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:36.858290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f973d-d1ea-41ae-978a-9384e43cc574 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:36.859583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 736/1453] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:01:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:36.865421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7a61ed-5c47-4971-9bba-580e6c6fa0a6 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:36.865861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7a61ed-5c47-4971-9bba-580e6c6fa0a6 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:36.865970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7a61ed-5c47-4971-9bba-580e6c6fa0a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:36.928789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:36.932899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ee7a61ed-5c47-4971-9bba-580e6c6fa0a6 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:36.934386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ee7a61ed-5c47-4971-9bba-580e6c6fa0a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:37.031429 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 718/1454] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:01:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:01:37.070200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4f973d-d1ea-41ae-978a-9384e43cc574 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:37.071998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-df4f973d-d1ea-41ae-978a-9384e43cc574 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:37.181644 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4f973d-d1ea-41ae-978a-9384e43cc574 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:37.233018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:37.262042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34783edc-9238-4861-a7e5-363ed1850c99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:37.262658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34783edc-9238-4861-a7e5-363ed1850c99 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:37.262939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34783edc-9238-4861-a7e5-363ed1850c99 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:37.263211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34783edc-9238-4861-a7e5-363ed1850c99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:37.264959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-34783edc-9238-4861-a7e5-363ed1850c99 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:37.383127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34783edc-9238-4861-a7e5-363ed1850c99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:37.384607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34783edc-9238-4861-a7e5-363ed1850c99 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:37.404343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 737/1455] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:39.608342 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec46044-670b-422c-8fae-cf9713adc92e None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:39.608856 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec46044-670b-422c-8fae-cf9713adc92e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:39.608856 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec46044-670b-422c-8fae-cf9713adc92e None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:39.610687 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 719/1456] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:01:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:39.617082 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85c4647-3a5b-4e74-ac84-70d8aea6d76d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:39.617665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85c4647-3a5b-4e74-ac84-70d8aea6d76d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:39.617791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85c4647-3a5b-4e74-ac84-70d8aea6d76d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:39.691402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f85c4647-3a5b-4e74-ac84-70d8aea6d76d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:39.692668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f85c4647-3a5b-4e74-ac84-70d8aea6d76d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:39.731992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:39.819932 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 738/1457] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:01:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:39.866799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec46044-670b-422c-8fae-cf9713adc92e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:39.868364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-cec46044-670b-422c-8fae-cf9713adc92e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:39.996313 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec46044-670b-422c-8fae-cf9713adc92e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:40.036659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:40.096203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd217954-9d04-4793-a02e-30a5aca9dccc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:40.097351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd217954-9d04-4793-a02e-30a5aca9dccc service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:40.097635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd217954-9d04-4793-a02e-30a5aca9dccc service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:40.097905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd217954-9d04-4793-a02e-30a5aca9dccc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:40.099567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-bd217954-9d04-4793-a02e-30a5aca9dccc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:40.231429 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd217954-9d04-4793-a02e-30a5aca9dccc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:40.232878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd217954-9d04-4793-a02e-30a5aca9dccc service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:40.256716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 720/1458] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:42.403723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a02f82-5ed9-4003-a05c-10b1cea6cb16 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:42.404323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a02f82-5ed9-4003-a05c-10b1cea6cb16 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:42.404323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a02f82-5ed9-4003-a05c-10b1cea6cb16 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:42.405674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 739/1459] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:01:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:42.411972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2327b69-36be-466d-99cf-54193982565b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:42.412476 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2327b69-36be-466d-99cf-54193982565b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:42.412578 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2327b69-36be-466d-99cf-54193982565b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:42.489643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b2327b69-36be-466d-99cf-54193982565b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:42.490874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b2327b69-36be-466d-99cf-54193982565b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:42.499807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:42.592155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 721/1460] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:01:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:42.629683 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a02f82-5ed9-4003-a05c-10b1cea6cb16 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:42.631196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f7a02f82-5ed9-4003-a05c-10b1cea6cb16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:42.735672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a02f82-5ed9-4003-a05c-10b1cea6cb16 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:42.790825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:42.809562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec06765-e030-452e-a242-27d5d176b39f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:42.810168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec06765-e030-452e-a242-27d5d176b39f service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:42.810445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec06765-e030-452e-a242-27d5d176b39f service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:42.810714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec06765-e030-452e-a242-27d5d176b39f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:42.812357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6ec06765-e030-452e-a242-27d5d176b39f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:42.919855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec06765-e030-452e-a242-27d5d176b39f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:42.921299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec06765-e030-452e-a242-27d5d176b39f service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:42.939187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 740/1461] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:44.998823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562adc39-9c39-42d3-aecc-18e120627856 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:44.999410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562adc39-9c39-42d3-aecc-18e120627856 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:44.999410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562adc39-9c39-42d3-aecc-18e120627856 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:45.000779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 722/1462] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:01:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:45.006147 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39adaa1e-501c-4c18-ac54-f6a5e0e9774c None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:45.006581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39adaa1e-501c-4c18-ac54-f6a5e0e9774c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:45.006686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39adaa1e-501c-4c18-ac54-f6a5e0e9774c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:45.072233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-39adaa1e-501c-4c18-ac54-f6a5e0e9774c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:45.073431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-39adaa1e-501c-4c18-ac54-f6a5e0e9774c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:45.113597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:45.171421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 741/1463] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:01:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:45.209847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562adc39-9c39-42d3-aecc-18e120627856 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:45.211379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-562adc39-9c39-42d3-aecc-18e120627856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:45.311320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562adc39-9c39-42d3-aecc-18e120627856 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:45.350648 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:45.389388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49157c3b-fdfe-4572-9ccd-ecc6b9ba64ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:45.390250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49157c3b-fdfe-4572-9ccd-ecc6b9ba64ab service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:45.390520 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49157c3b-fdfe-4572-9ccd-ecc6b9ba64ab service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:45.390787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49157c3b-fdfe-4572-9ccd-ecc6b9ba64ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:45.392419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-49157c3b-fdfe-4572-9ccd-ecc6b9ba64ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:45.504436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49157c3b-fdfe-4572-9ccd-ecc6b9ba64ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:45.505877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49157c3b-fdfe-4572-9ccd-ecc6b9ba64ab service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:45.524652 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 723/1464] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:47.547968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2833ea54-3253-4513-ac65-8163d00615e5 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:47.548542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2833ea54-3253-4513-ac65-8163d00615e5 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:47.548542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2833ea54-3253-4513-ac65-8163d00615e5 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:47.549852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 742/1465] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:01:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:47.555934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18634c-f0e6-48c9-8cd8-d32e1665976a None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:47.556344 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18634c-f0e6-48c9-8cd8-d32e1665976a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:47.556492 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18634c-f0e6-48c9-8cd8-d32e1665976a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:47.623268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-2b18634c-f0e6-48c9-8cd8-d32e1665976a None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:47.624489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2b18634c-f0e6-48c9-8cd8-d32e1665976a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:47.642292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:47.719894 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 724/1466] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:01:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:47.756791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2833ea54-3253-4513-ac65-8163d00615e5 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:47.758300 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2833ea54-3253-4513-ac65-8163d00615e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:47.860276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2833ea54-3253-4513-ac65-8163d00615e5 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:47.926203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:47.933568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d1cd2e-667d-44b6-a046-824ee54bf6cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:47.934174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d1cd2e-667d-44b6-a046-824ee54bf6cd service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:47.934446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d1cd2e-667d-44b6-a046-824ee54bf6cd service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:47.934711 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d1cd2e-667d-44b6-a046-824ee54bf6cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:47.936347 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-27d1cd2e-667d-44b6-a046-824ee54bf6cd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:48.044983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d1cd2e-667d-44b6-a046-824ee54bf6cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:48.046308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d1cd2e-667d-44b6-a046-824ee54bf6cd service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:48.064561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 743/1467] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:01:50.072288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc3836a-247f-467c-adb5-59714cbfd207 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:50.072858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc3836a-247f-467c-adb5-59714cbfd207 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:50.072858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc3836a-247f-467c-adb5-59714cbfd207 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:50.074179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 725/1468] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:01:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:01:50.079319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab24e84-3346-4af4-b039-111ce0204066 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:50.079762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab24e84-3346-4af4-b039-111ce0204066 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:50.079872 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab24e84-3346-4af4-b039-111ce0204066 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:50.144364 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-7ab24e84-3346-4af4-b039-111ce0204066 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:01:50.145675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7ab24e84-3346-4af4-b039-111ce0204066 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:50.200206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:50.254612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 744/1469] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:01:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:01:50.291589 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc3836a-247f-467c-adb5-59714cbfd207 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:01:50.293109 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ebc3836a-247f-467c-adb5-59714cbfd207 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:50.396653 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc3836a-247f-467c-adb5-59714cbfd207 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:01:50.443895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:01:50.469809 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cfb4f1-f3b1-4aa8-a904-cdf75ec5223c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:01:50.470409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cfb4f1-f3b1-4aa8-a904-cdf75ec5223c service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:01:50.470681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cfb4f1-f3b1-4aa8-a904-cdf75ec5223c service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:01:50.470944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cfb4f1-f3b1-4aa8-a904-cdf75ec5223c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:01:50.472588 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f2cfb4f1-f3b1-4aa8-a904-cdf75ec5223c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:01:50.589317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cfb4f1-f3b1-4aa8-a904-cdf75ec5223c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:01:50.590749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2cfb4f1-f3b1-4aa8-a904-cdf75ec5223c service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:01:50.609180 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 726/1470] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:01:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:02:00.250159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=22 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:00.606869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=66 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:10.132924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa46733-e978-40dd-a845-a62602363eee None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:10.133530 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa46733-e978-40dd-a845-a62602363eee None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:10.133530 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa46733-e978-40dd-a845-a62602363eee None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:10.134838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 745/1471] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:02:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:02:10.140264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0b0a3-5dc9-48e1-b896-864539359e27 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:10.140738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0b0a3-5dc9-48e1-b896-864539359e27 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:10.140839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0b0a3-5dc9-48e1-b896-864539359e27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:10.218886 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-4ef0b0a3-5dc9-48e1-b896-864539359e27 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:02:10.220120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-4ef0b0a3-5dc9-48e1-b896-864539359e27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:10.322754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 727/1472] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:02:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:02:10.406891 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20bd4df-746d-4a74-86c7-c2cd4307b224 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:10.407175 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20bd4df-746d-4a74-86c7-c2cd4307b224 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:10.407471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20bd4df-746d-4a74-86c7-c2cd4307b224 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:10.408797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 746/1473] 10.222.0.42 () {54 vars in 966 bytes} [Sun Jul 31 01:02:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:02:10.413523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff74107c-7d41-4af1-ac59-b0667383d7c1 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:10.413811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff74107c-7d41-4af1-ac59-b0667383d7c1 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:10.414089 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff74107c-7d41-4af1-ac59-b0667383d7c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:10.497021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ff74107c-7d41-4af1-ac59-b0667383d7c1 None None] MFA Rules not processed for user `4160039561434e51a87c6c7a019f2dff`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:02:10.498220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ff74107c-7d41-4af1-ac59-b0667383d7c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:10.518535 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:10.580852 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 728/1474] 10.222.0.42 () {58 vars in 1063 bytes} [Sun Jul 31 01:02:10 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:02:10.589176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20bd4df-746d-4a74-86c7-c2cd4307b224 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:02:10.590746 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d20bd4df-746d-4a74-86c7-c2cd4307b224 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:10.699660 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20bd4df-746d-4a74-86c7-c2cd4307b224 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:02:10.774713 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4627ed3e-85a8-4548-877d-bae19615c4fc service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4160039561434e51a87c6c7a019f2dff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:02:10.775424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4627ed3e-85a8-4548-877d-bae19615c4fc service glance] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:10.775702 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4627ed3e-85a8-4548-877d-bae19615c4fc service glance] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:10.775968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4627ed3e-85a8-4548-877d-bae19615c4fc service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:10.777595 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4627ed3e-85a8-4548-877d-bae19615c4fc service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:10.835447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:10.892704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4627ed3e-85a8-4548-877d-bae19615c4fc service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:02:10.894132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4627ed3e-85a8-4548-877d-bae19615c4fc service glance] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:02:10.912749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 747/1475] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:02:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:02:13.002936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b269a0-afd9-480e-948b-2ecc8d0c0eba None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:13.003524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b269a0-afd9-480e-948b-2ecc8d0c0eba None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:13.003524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b269a0-afd9-480e-948b-2ecc8d0c0eba None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:13.004896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 729/1476] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:02:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:02:13.010111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d37b93-9ab8-451c-9b20-5b6b775a0dc5 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:13.010563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d37b93-9ab8-451c-9b20-5b6b775a0dc5 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:13.010666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d37b93-9ab8-451c-9b20-5b6b775a0dc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:13.075461 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b4d37b93-9ab8-451c-9b20-5b6b775a0dc5 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:02:13.076658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b4d37b93-9ab8-451c-9b20-5b6b775a0dc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:13.172551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 748/1477] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:02:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:02:13.195546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b269a0-afd9-480e-948b-2ecc8d0c0eba None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:02:13.197065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-08b269a0-afd9-480e-948b-2ecc8d0c0eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:13.329275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b269a0-afd9-480e-948b-2ecc8d0c0eba None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:02:13.378003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:13.422718 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51290ec4-3c4a-4d01-9262-788a97750824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:02:13.423453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51290ec4-3c4a-4d01-9262-788a97750824 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:13.423859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51290ec4-3c4a-4d01-9262-788a97750824 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:13.424105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51290ec4-3c4a-4d01-9262-788a97750824 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:13.424795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51290ec4-3c4a-4d01-9262-788a97750824 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:02:13.426525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: WARNING py.warnings [None req-51290ec4-3c4a-4d01-9262-788a97750824 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:02:13.426525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: warnings.warn(msg) Jul 31 01:02:13.426525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: Jul 31 01:02:13.428047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51290ec4-3c4a-4d01-9262-788a97750824 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:02:13.446935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 730/1478] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Jul 31 01:02:13 2022] GET /identity/v3/projects => generated 2553 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:02:15.464048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6549224-11df-46ee-a436-1939d5ca2d0b None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:15.464560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6549224-11df-46ee-a436-1939d5ca2d0b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:15.464656 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6549224-11df-46ee-a436-1939d5ca2d0b None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:15.465927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 749/1479] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:02:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:02:15.471045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5901a5ca-f85b-45cb-9c66-a37e732a4c9f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:15.471500 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5901a5ca-f85b-45cb-9c66-a37e732a4c9f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:15.471629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5901a5ca-f85b-45cb-9c66-a37e732a4c9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:15.537859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-5901a5ca-f85b-45cb-9c66-a37e732a4c9f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:02:15.539072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5901a5ca-f85b-45cb-9c66-a37e732a4c9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:15.641082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 731/1480] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:02:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:02:15.685009 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6549224-11df-46ee-a436-1939d5ca2d0b None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:02:15.686553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f6549224-11df-46ee-a436-1939d5ca2d0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:15.696334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:15.832960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6549224-11df-46ee-a436-1939d5ca2d0b None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:02:15.912493 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:15.951641 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0704a4c6-dd31-4702-aa68-f609588cbe7f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:02:15.952173 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0704a4c6-dd31-4702-aa68-f609588cbe7f service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:15.952449 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0704a4c6-dd31-4702-aa68-f609588cbe7f service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:15.952718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0704a4c6-dd31-4702-aa68-f609588cbe7f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:15.954329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0704a4c6-dd31-4702-aa68-f609588cbe7f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:16.041878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0704a4c6-dd31-4702-aa68-f609588cbe7f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:02:16.043491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0704a4c6-dd31-4702-aa68-f609588cbe7f service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:02:16.061471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 750/1481] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:02:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:02:18.257175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b408989-79d8-41c9-809f-01abbc9862f1 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:18.257745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b408989-79d8-41c9-809f-01abbc9862f1 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:18.257745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b408989-79d8-41c9-809f-01abbc9862f1 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:18.259100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 732/1482] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:02:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:02:18.264442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfabdec6-e321-47b5-af1a-35f72cbae8d2 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:18.264897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfabdec6-e321-47b5-af1a-35f72cbae8d2 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:18.264993 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfabdec6-e321-47b5-af1a-35f72cbae8d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:18.330256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-cfabdec6-e321-47b5-af1a-35f72cbae8d2 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:02:18.331466 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-cfabdec6-e321-47b5-af1a-35f72cbae8d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:18.377050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:18.427753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 751/1483] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:02:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:02:18.463887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b408989-79d8-41c9-809f-01abbc9862f1 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:02:18.465589 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8b408989-79d8-41c9-809f-01abbc9862f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:18.489548 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:18.574497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b408989-79d8-41c9-809f-01abbc9862f1 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:02:18.647473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cb2fce-e6d2-4d3f-adf6-c0fe03346d84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:02:18.648083 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cb2fce-e6d2-4d3f-adf6-c0fe03346d84 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:18.648357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cb2fce-e6d2-4d3f-adf6-c0fe03346d84 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:18.648627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cb2fce-e6d2-4d3f-adf6-c0fe03346d84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:18.650225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f4cb2fce-e6d2-4d3f-adf6-c0fe03346d84 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:18.723455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:18.760293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4cb2fce-e6d2-4d3f-adf6-c0fe03346d84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:02:18.761759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4cb2fce-e6d2-4d3f-adf6-c0fe03346d84 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:02:18.780073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 733/1484] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:02:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:02:20.874811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b354a4-4457-4d06-85e3-8fa1361674fb None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:20.875338 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b354a4-4457-4d06-85e3-8fa1361674fb None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:20.875338 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b354a4-4457-4d06-85e3-8fa1361674fb None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:20.876629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 752/1485] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:02:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:02:20.882588 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279d0b89-8d1c-48de-8eda-470d9e4fb80e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:20.883038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279d0b89-8d1c-48de-8eda-470d9e4fb80e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:20.883202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279d0b89-8d1c-48de-8eda-470d9e4fb80e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:20.948759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-279d0b89-8d1c-48de-8eda-470d9e4fb80e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:02:20.949976 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-279d0b89-8d1c-48de-8eda-470d9e4fb80e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:21.055234 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 734/1486] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:02:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:02:21.094030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b354a4-4457-4d06-85e3-8fa1361674fb None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:02:21.095579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-91b354a4-4457-4d06-85e3-8fa1361674fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:21.198277 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b354a4-4457-4d06-85e3-8fa1361674fb None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:02:21.269810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:21.272444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814fa5b9-f83e-413b-8c7a-9d68760f9104 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:02:21.273298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814fa5b9-f83e-413b-8c7a-9d68760f9104 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:21.273599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814fa5b9-f83e-413b-8c7a-9d68760f9104 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:21.274017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814fa5b9-f83e-413b-8c7a-9d68760f9104 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:21.275840 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-814fa5b9-f83e-413b-8c7a-9d68760f9104 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:21.386497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814fa5b9-f83e-413b-8c7a-9d68760f9104 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:02:21.387950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814fa5b9-f83e-413b-8c7a-9d68760f9104 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:02:21.406208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 753/1487] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:02:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:02:23.653506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c2a4ff-3d7f-4714-98fa-eb4e60dffdfa None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:23.654158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c2a4ff-3d7f-4714-98fa-eb4e60dffdfa None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:23.654158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c2a4ff-3d7f-4714-98fa-eb4e60dffdfa None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:23.655413 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 735/1488] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:02:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:02:23.661749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657da2cb-b42b-4fdc-be09-048a4ee562e4 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:23.662170 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657da2cb-b42b-4fdc-be09-048a4ee562e4 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:23.662290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657da2cb-b42b-4fdc-be09-048a4ee562e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:23.728578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-657da2cb-b42b-4fdc-be09-048a4ee562e4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:02:23.729679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-657da2cb-b42b-4fdc-be09-048a4ee562e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:23.780450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:23.824324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 754/1489] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:02:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:02:23.866104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c2a4ff-3d7f-4714-98fa-eb4e60dffdfa None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:02:23.867695 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-95c2a4ff-3d7f-4714-98fa-eb4e60dffdfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:23.903754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:23.972495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c2a4ff-3d7f-4714-98fa-eb4e60dffdfa None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:02:24.044683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88e6f48-672f-4ce0-9cb9-7d35a6829ecb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:02:24.045201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88e6f48-672f-4ce0-9cb9-7d35a6829ecb service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:24.045478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88e6f48-672f-4ce0-9cb9-7d35a6829ecb service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:24.045748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88e6f48-672f-4ce0-9cb9-7d35a6829ecb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:24.047567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e88e6f48-672f-4ce0-9cb9-7d35a6829ecb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:24.126712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:24.156939 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88e6f48-672f-4ce0-9cb9-7d35a6829ecb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:02:24.158373 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88e6f48-672f-4ce0-9cb9-7d35a6829ecb service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:02:24.176473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 736/1490] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:02:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:02:26.402103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997966ed-2497-4ae3-80f5-501af4e62a09 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:26.402668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997966ed-2497-4ae3-80f5-501af4e62a09 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:26.402668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997966ed-2497-4ae3-80f5-501af4e62a09 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:26.404048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 755/1491] 10.222.0.42 () {54 vars in 926 bytes} [Sun Jul 31 01:02:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:02:26.409275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717eb5ad-497e-4c01-8f34-34dde742e084 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:26.409738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717eb5ad-497e-4c01-8f34-34dde742e084 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:26.409834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717eb5ad-497e-4c01-8f34-34dde742e084 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:26.474122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-717eb5ad-497e-4c01-8f34-34dde742e084 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:02:26.475306 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-717eb5ad-497e-4c01-8f34-34dde742e084 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:26.572330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 737/1492] 10.222.0.42 () {58 vars in 1023 bytes} [Sun Jul 31 01:02:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:02:26.612560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997966ed-2497-4ae3-80f5-501af4e62a09 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:02:26.614090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-997966ed-2497-4ae3-80f5-501af4e62a09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:26.716830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997966ed-2497-4ae3-80f5-501af4e62a09 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:02:26.788748 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a39b5f3-9497-4d08-a3cf-23883f12097f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:02:26.789366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a39b5f3-9497-4d08-a3cf-23883f12097f service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:02:26.789642 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a39b5f3-9497-4d08-a3cf-23883f12097f service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:02:26.789911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a39b5f3-9497-4d08-a3cf-23883f12097f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:02:26.791543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9a39b5f3-9497-4d08-a3cf-23883f12097f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:02:26.800608 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:26.899032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a39b5f3-9497-4d08-a3cf-23883f12097f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:02:26.900500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a39b5f3-9497-4d08-a3cf-23883f12097f service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:02:26.918169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 756/1493] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:02:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:02:36.570417 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=82 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:02:36.915943 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=50 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:04:05.099811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487ddc4e-21ae-4f89-a67d-771408bb29a7 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:05.100420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487ddc4e-21ae-4f89-a67d-771408bb29a7 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:05.100420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487ddc4e-21ae-4f89-a67d-771408bb29a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:05.178304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-487ddc4e-21ae-4f89-a67d-771408bb29a7 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:04:05.180139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-487ddc4e-21ae-4f89-a67d-771408bb29a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:05.276969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 738/1494] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:04:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:04:05.285273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849d014a-a415-4830-a20b-cda358ede042 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:05.285774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849d014a-a415-4830-a20b-cda358ede042 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:05.285880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849d014a-a415-4830-a20b-cda358ede042 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:05.363084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-849d014a-a415-4830-a20b-cda358ede042 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:04:05.364238 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-849d014a-a415-4830-a20b-cda358ede042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:05.458795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 757/1495] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:04:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:04:05.464154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487ddc4e-21ae-4f89-a67d-771408bb29a7 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:04:05.465352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-487ddc4e-21ae-4f89-a67d-771408bb29a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:05.477342 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487ddc4e-21ae-4f89-a67d-771408bb29a7 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:04:05.539547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:04:05.573448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a5e18e-7b4a-4517-8cb5-34c3bee40570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:04:05.574055 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a5e18e-7b4a-4517-8cb5-34c3bee40570 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:05.574329 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a5e18e-7b4a-4517-8cb5-34c3bee40570 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:05.574596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a5e18e-7b4a-4517-8cb5-34c3bee40570 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:05.575273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a5e18e-7b4a-4517-8cb5-34c3bee40570 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:04:05.575618 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a5e18e-7b4a-4517-8cb5-34c3bee40570 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:04:05.576621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: WARNING py.warnings [None req-c8a5e18e-7b4a-4517-8cb5-34c3bee40570 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:04:05.576621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: warnings.warn(msg) Jul 31 01:04:05.576621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: Jul 31 01:04:05.577502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a5e18e-7b4a-4517-8cb5-34c3bee40570 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:04:05.589053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 739/1496] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:04:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:04:05.597602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849d014a-a415-4830-a20b-cda358ede042 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:04:05.607411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849d014a-a415-4830-a20b-cda358ede042 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:04:05.668798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:04:05.701790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48c7675-ea77-4e15-aa4e-b17e010a3815 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:04:05.702403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48c7675-ea77-4e15-aa4e-b17e010a3815 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:05.702678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48c7675-ea77-4e15-aa4e-b17e010a3815 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:05.702944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48c7675-ea77-4e15-aa4e-b17e010a3815 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:05.703666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48c7675-ea77-4e15-aa4e-b17e010a3815 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:04:05.705379 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: WARNING py.warnings [None req-b48c7675-ea77-4e15-aa4e-b17e010a3815 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:04:05.705379 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: warnings.warn(msg) Jul 31 01:04:05.705379 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: Jul 31 01:04:05.706573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48c7675-ea77-4e15-aa4e-b17e010a3815 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:04:05.761302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 758/1497] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:04:05 2022] POST /identity/v3/projects => generated 377 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:04:05.765521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a5e18e-7b4a-4517-8cb5-34c3bee40570 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:04:05.767032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c8a5e18e-7b4a-4517-8cb5-34c3bee40570 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:05.800527 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a5e18e-7b4a-4517-8cb5-34c3bee40570 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:04:05.894684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f03648-021b-4951-bce1-7aea0b398c2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:04:05.895323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f03648-021b-4951-bce1-7aea0b398c2a admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:05.895743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f03648-021b-4951-bce1-7aea0b398c2a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:05.896020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f03648-021b-4951-bce1-7aea0b398c2a admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:05.896752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f03648-021b-4951-bce1-7aea0b398c2a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:04:05.898383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: WARNING py.warnings [None req-70f03648-021b-4951-bce1-7aea0b398c2a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:04:05.898383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: warnings.warn(msg) Jul 31 01:04:05.898383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: Jul 31 01:04:05.899608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f03648-021b-4951-bce1-7aea0b398c2a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:04:05.940295 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 740/1498] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:04:05 2022] POST /identity/v3/users => generated 289 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:04:05.944926 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48c7675-ea77-4e15-aa4e-b17e010a3815 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:04:05.946417 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b48c7675-ea77-4e15-aa4e-b17e010a3815 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:06.006771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48c7675-ea77-4e15-aa4e-b17e010a3815 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:04:06.088274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:04:06.099145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c2be00-c3f3-4042-aebb-28105100c2e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:04:06.099780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c2be00-c3f3-4042-aebb-28105100c2e8 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:06.100054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c2be00-c3f3-4042-aebb-28105100c2e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:06.100320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c2be00-c3f3-4042-aebb-28105100c2e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:06.101027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c2be00-c3f3-4042-aebb-28105100c2e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:04:06.102031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: WARNING py.warnings [None req-49c2be00-c3f3-4042-aebb-28105100c2e8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:04:06.102031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: warnings.warn(msg) Jul 31 01:04:06.102031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: Jul 31 01:04:06.102909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c2be00-c3f3-4042-aebb-28105100c2e8 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:04:06.115033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 759/1499] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:04:05 2022] GET /identity/v3/roles => generated 1420 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:04:06.119115 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f03648-021b-4951-bce1-7aea0b398c2a admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:04:06.120711 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-70f03648-021b-4951-bce1-7aea0b398c2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:06.133619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f03648-021b-4951-bce1-7aea0b398c2a admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:04:06.160358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:04:06.230850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c7b78f-42ac-48e3-9731-b421a2b7a0ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:04:06.231397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c7b78f-42ac-48e3-9731-b421a2b7a0ba admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:06.231677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c7b78f-42ac-48e3-9731-b421a2b7a0ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:06.231942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c7b78f-42ac-48e3-9731-b421a2b7a0ba admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:06.232626 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c7b78f-42ac-48e3-9731-b421a2b7a0ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:04:06.233638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: WARNING py.warnings [None req-22c7b78f-42ac-48e3-9731-b421a2b7a0ba admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:04:06.233638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: warnings.warn(msg) Jul 31 01:04:06.233638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: Jul 31 01:04:06.234514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c7b78f-42ac-48e3-9731-b421a2b7a0ba admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:04:06.248115 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 741/1500] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:04:06.252330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c2be00-c3f3-4042-aebb-28105100c2e8 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:04:06.253847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-49c2be00-c3f3-4042-aebb-28105100c2e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:06.288039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c2be00-c3f3-4042-aebb-28105100c2e8 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:04:06.380957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3633dde-c1ec-40da-b48d-4e4197368c74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:04:06.381620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3633dde-c1ec-40da-b48d-4e4197368c74 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:06.382470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3633dde-c1ec-40da-b48d-4e4197368c74 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:06.382753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3633dde-c1ec-40da-b48d-4e4197368c74 admin admin] PATH_INFO: `/v3/projects/262afe1a356247a8840e916f7ef03251/users/60f7a8611e554fbb9013a46bec54f4b1/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:06.404729 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3633dde-c1ec-40da-b48d-4e4197368c74 admin admin] RBAC: Authorizing `identity:create_grant(project_id=262afe1a356247a8840e916f7ef03251, user_id=60f7a8611e554fbb9013a46bec54f4b1, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:04:06.405761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: WARNING py.warnings [None req-b3633dde-c1ec-40da-b48d-4e4197368c74 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:04:06.405761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: warnings.warn(msg) Jul 31 01:04:06.405761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: Jul 31 01:04:06.407849 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3633dde-c1ec-40da-b48d-4e4197368c74 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:04:06.433449 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 760/1501] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:04:06 2022] PUT /identity/v3/projects/262afe1a356247a8840e916f7ef03251/users/60f7a8611e554fbb9013a46bec54f4b1/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:04:06.446432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e9c96c-6ca9-4083-84a7-0699c6fe1d5f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:06.446723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e9c96c-6ca9-4083-84a7-0699c6fe1d5f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:06.447003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e9c96c-6ca9-4083-84a7-0699c6fe1d5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:06.502161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-13e9c96c-6ca9-4083-84a7-0699c6fe1d5f None None] MFA Rules not processed for user `60f7a8611e554fbb9013a46bec54f4b1`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:04:06.503304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-13e9c96c-6ca9-4083-84a7-0699c6fe1d5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:06.583928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 742/1502] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:04:06 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:04:06.597199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90650b8b-451c-4f1d-8ed6-bf6fb2a2e033 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:06.597485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90650b8b-451c-4f1d-8ed6-bf6fb2a2e033 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:06.597761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90650b8b-451c-4f1d-8ed6-bf6fb2a2e033 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:06.599112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 761/1503] 10.222.0.42 () {54 vars in 964 bytes} [Sun Jul 31 01:04:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:04:06.604563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a956e3b9-a104-4d7a-bba3-147547cdf6fb None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:06.604859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a956e3b9-a104-4d7a-bba3-147547cdf6fb None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:06.605156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a956e3b9-a104-4d7a-bba3-147547cdf6fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:06.637444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:04:06.681508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a956e3b9-a104-4d7a-bba3-147547cdf6fb None None] MFA Rules not processed for user `9c8338bf95d34b228176b556afd93155`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:04:06.682955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a956e3b9-a104-4d7a-bba3-147547cdf6fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:06.756850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 743/1504] 10.222.0.42 () {58 vars in 1061 bytes} [Sun Jul 31 01:04:06 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:04:06.764653 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90650b8b-451c-4f1d-8ed6-bf6fb2a2e033 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:04:06.766204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-90650b8b-451c-4f1d-8ed6-bf6fb2a2e033 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:06.815847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:04:06.872225 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90650b8b-451c-4f1d-8ed6-bf6fb2a2e033 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:04:06.945132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a1716f-7c23-4b08-b9ab-a0f074f23e38 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:04:06.945740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a1716f-7c23-4b08-b9ab-a0f074f23e38 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:06.946016 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a1716f-7c23-4b08-b9ab-a0f074f23e38 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:06.946287 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a1716f-7c23-4b08-b9ab-a0f074f23e38 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:06.947915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-39a1716f-7c23-4b08-b9ab-a0f074f23e38 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:07.044360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a1716f-7c23-4b08-b9ab-a0f074f23e38 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:04:07.045704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a1716f-7c23-4b08-b9ab-a0f074f23e38 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:04:07.049150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:04:07.064065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 762/1505] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:04:06 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:04:07.110527 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34800fbd-79a0-44a6-9ac2-90cb41db6f9f None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:07.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34800fbd-79a0-44a6-9ac2-90cb41db6f9f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:07.111081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34800fbd-79a0-44a6-9ac2-90cb41db6f9f None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:07.112838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 744/1506] 10.222.0.42 () {52 vars in 830 bytes} [Sun Jul 31 01:04:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:04:07.116592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a1716f-7c23-4b08-b9ab-a0f074f23e38 service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:04:07.118177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-39a1716f-7c23-4b08-b9ab-a0f074f23e38 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:07.223604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a1716f-7c23-4b08-b9ab-a0f074f23e38 service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:04:07.317288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:04:07.317930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:07.318079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:07.318343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] PATH_INFO: `/v3/users/60f7a8611e554fbb9013a46bec54f4b1` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:07.330817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] RBAC: Authorizing `identity:delete_user(user_id=60f7a8611e554fbb9013a46bec54f4b1)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:04:07.331864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: WARNING py.warnings [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:04:07.331864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: warnings.warn(msg) Jul 31 01:04:07.331864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: Jul 31 01:04:07.333072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:04:07.346007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:04:07.402655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '60f7a8611e554fbb9013a46bec54f4b1'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:04:07.403165 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '60f7a8611e554fbb9013a46bec54f4b1'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:04:07.412258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '60f7a8611e554fbb9013a46bec54f4b1'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:04:07.434801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '60f7a8611e554fbb9013a46bec54f4b1'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:04:07.449186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '60f7a8611e554fbb9013a46bec54f4b1'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:04:07.454628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 763/1507] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:04:07 2022] DELETE /identity/v3/users/60f7a8611e554fbb9013a46bec54f4b1 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:04:07.459120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79eca50b-db37-4374-8ccf-1a00b4a428c8 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:07.459627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79eca50b-db37-4374-8ccf-1a00b4a428c8 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:07.459731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79eca50b-db37-4374-8ccf-1a00b4a428c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:07.515723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-79eca50b-db37-4374-8ccf-1a00b4a428c8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:04:07.516905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-79eca50b-db37-4374-8ccf-1a00b4a428c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:07.601942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:04:07.615290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 745/1508] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:04:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:04:07.624864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:04:07.626091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:07.726201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd82d6e-31a1-443a-bb9f-fd9548391e67 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:04:07.780544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:04:07.800462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9a46cb-72f7-4b55-b7c7-e26ba90d6c03 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:04:07.801107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9a46cb-72f7-4b55-b7c7-e26ba90d6c03 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:07.801385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9a46cb-72f7-4b55-b7c7-e26ba90d6c03 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:07.801657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9a46cb-72f7-4b55-b7c7-e26ba90d6c03 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:07.803267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4a9a46cb-72f7-4b55-b7c7-e26ba90d6c03 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:07.911974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9a46cb-72f7-4b55-b7c7-e26ba90d6c03 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:04:07.913432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9a46cb-72f7-4b55-b7c7-e26ba90d6c03 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:04:07.932719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 764/1509] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:04:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:04:08.290461 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79eca50b-db37-4374-8ccf-1a00b4a428c8 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:04:08.291958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-79eca50b-db37-4374-8ccf-1a00b4a428c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:04:08.303727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79eca50b-db37-4374-8ccf-1a00b4a428c8 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:04:08.408689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:04:08.409339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:04:08.409613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:04:08.409888 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] PATH_INFO: `/v3/projects/262afe1a356247a8840e916f7ef03251` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:04:08.421218 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] RBAC: Authorizing `identity:delete_project(project_id=262afe1a356247a8840e916f7ef03251)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:04:08.422189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: WARNING py.warnings [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:04:08.422189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: warnings.warn(msg) Jul 31 01:04:08.422189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: Jul 31 01:04:08.423423 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:04:08.484530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '262afe1a356247a8840e916f7ef03251', 'request_id': 'req-03a096a8-dc1c-4af5-8114-c21128273c28'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:04:08.498895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '262afe1a356247a8840e916f7ef03251', 'request_id': 'req-03a096a8-dc1c-4af5-8114-c21128273c28'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:04:08.504835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '262afe1a356247a8840e916f7ef03251', 'request_id': 'req-03a096a8-dc1c-4af5-8114-c21128273c28'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:04:08.532201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] The token cache is being invalidate because project 262afe1a356247a8840e916f7ef03251 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:04:08.532533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:04:08.532928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-03a096a8-dc1c-4af5-8114-c21128273c28 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:04:08.534660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 746/1510] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:04:08 2022] DELETE /identity/v3/projects/262afe1a356247a8840e916f7ef03251 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:04:17.930179 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=62 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:04:18.529942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:05:49.927793 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddddf577-15d9-4f90-9653-7f3383daf4d5 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:05:49.929049 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddddf577-15d9-4f90-9653-7f3383daf4d5 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:05:49.929049 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddddf577-15d9-4f90-9653-7f3383daf4d5 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:05:49.930415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 765/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 31 01:05:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:05:49.936060 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79683a9-dc9b-48b1-94be-4b519ab45d90 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:05:49.936907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79683a9-dc9b-48b1-94be-4b519ab45d90 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:05:49.937085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79683a9-dc9b-48b1-94be-4b519ab45d90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:05:50.039098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c79683a9-dc9b-48b1-94be-4b519ab45d90 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:05:50.040387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c79683a9-dc9b-48b1-94be-4b519ab45d90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:05:50.159248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 747/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 31 01:05:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:05:50.319433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddddf577-15d9-4f90-9653-7f3383daf4d5 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:05:50.322609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ddddf577-15d9-4f90-9653-7f3383daf4d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:05:50.539675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddddf577-15d9-4f90-9653-7f3383daf4d5 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:05:50.547771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3924640a-f237-4962-a3eb-9eb905a925ad service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:05:50.548420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3924640a-f237-4962-a3eb-9eb905a925ad service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:05:50.548703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3924640a-f237-4962-a3eb-9eb905a925ad service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:05:50.548985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3924640a-f237-4962-a3eb-9eb905a925ad service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:05:50.559252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3924640a-f237-4962-a3eb-9eb905a925ad service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:05:50.561539 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3924640a-f237-4962-a3eb-9eb905a925ad service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:05:50.573345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 766/1513] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:05:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:05:51.502865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf0824f-d408-436d-99d6-52c912bf0998 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:05:51.503696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf0824f-d408-436d-99d6-52c912bf0998 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:05:51.503696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf0824f-d408-436d-99d6-52c912bf0998 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:05:51.505104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 748/1514] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 31 01:05:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:05:51.511115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6311f-9aeb-4b3e-8493-712c00a2858e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:05:51.511819 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6311f-9aeb-4b3e-8493-712c00a2858e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:05:51.511819 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6311f-9aeb-4b3e-8493-712c00a2858e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:05:51.534064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-12b6311f-9aeb-4b3e-8493-712c00a2858e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:05:51.535330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-12b6311f-9aeb-4b3e-8493-712c00a2858e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:05:51.543959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 767/1515] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 31 01:05:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:05:51.725571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf0824f-d408-436d-99d6-52c912bf0998 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:05:51.742338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf0824f-d408-436d-99d6-52c912bf0998 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:05:51.748003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd964e7a-58b2-4f83-a023-f9306c4fdea4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:05:51.748633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd964e7a-58b2-4f83-a023-f9306c4fdea4 service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:05:51.748916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd964e7a-58b2-4f83-a023-f9306c4fdea4 service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:05:51.749193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd964e7a-58b2-4f83-a023-f9306c4fdea4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:05:51.759017 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd964e7a-58b2-4f83-a023-f9306c4fdea4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:05:51.760595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd964e7a-58b2-4f83-a023-f9306c4fdea4 service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:05:51.771023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 749/1516] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:05:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:05:52.156606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c60aa-7615-4d54-93a0-25c7b5ee863b None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:05:52.156887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c60aa-7615-4d54-93a0-25c7b5ee863b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:05:52.157169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c60aa-7615-4d54-93a0-25c7b5ee863b None None] PATH_INFO: `/v3` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:05:52.158497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 768/1517] 10.222.0.64 () {54 vars in 925 bytes} [Sun Jul 31 01:05:52 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:05:52.165356 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a87a238-6d9f-4cfd-b09c-893f748e5d78 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:05:52.165641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a87a238-6d9f-4cfd-b09c-893f748e5d78 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:05:52.165919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a87a238-6d9f-4cfd-b09c-893f748e5d78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:05:52.231732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-5a87a238-6d9f-4cfd-b09c-893f748e5d78 None None] MFA Rules not processed for user `1d562ccd50e4492281a9f435f809d49b`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:05:52.232947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5a87a238-6d9f-4cfd-b09c-893f748e5d78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:05:52.250915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:05:52.316199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 750/1518] 10.222.0.64 () {58 vars in 1015 bytes} [Sun Jul 31 01:05:52 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:06:01.341358 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a0c327-5d88-458e-a1a2-9b1e11dbed4f None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:01.342228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a0c327-5d88-458e-a1a2-9b1e11dbed4f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:01.342228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a0c327-5d88-458e-a1a2-9b1e11dbed4f None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:01.344201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 769/1519] 10.222.0.42 () {54 vars in 961 bytes} [Sun Jul 31 01:06:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:06:01.350032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af39c494-823f-41db-bf15-9bd6be9b9a47 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:01.350905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af39c494-823f-41db-bf15-9bd6be9b9a47 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:01.350905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af39c494-823f-41db-bf15-9bd6be9b9a47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:01.372043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-af39c494-823f-41db-bf15-9bd6be9b9a47 None None] MFA Rules not processed for user `1d562ccd50e4492281a9f435f809d49b`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:06:01.373266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-af39c494-823f-41db-bf15-9bd6be9b9a47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:06:01.382520 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 751/1520] 10.222.0.42 () {58 vars in 1058 bytes} [Sun Jul 31 01:06:01 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:06:01.390798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a0c327-5d88-458e-a1a2-9b1e11dbed4f None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:06:01.405364 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a0c327-5d88-458e-a1a2-9b1e11dbed4f None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:06:01.411078 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a492cf4-73bc-4e5b-8177-3fb45254d152 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:06:01.411938 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a492cf4-73bc-4e5b-8177-3fb45254d152 service placement] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:01.412297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a492cf4-73bc-4e5b-8177-3fb45254d152 service placement] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a492cf4-73bc-4e5b-8177-3fb45254d152 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:01.422442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a492cf4-73bc-4e5b-8177-3fb45254d152 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:06:01.424202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a492cf4-73bc-4e5b-8177-3fb45254d152 service placement] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:06:01.436729 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 770/1521] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:06:01 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:06:03.056901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d331fa-4d22-4f4e-a37e-3b31fe4d001a None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:03.057740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d331fa-4d22-4f4e-a37e-3b31fe4d001a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:03.057740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d331fa-4d22-4f4e-a37e-3b31fe4d001a None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:03.059300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 752/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 31 01:06:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:06:03.065822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a140e8-c7f1-47e7-94b2-d212002a7dfc None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:03.066404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a140e8-c7f1-47e7-94b2-d212002a7dfc None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:03.066804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a140e8-c7f1-47e7-94b2-d212002a7dfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:03.091954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-12a140e8-c7f1-47e7-94b2-d212002a7dfc None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:06:03.093372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-12a140e8-c7f1-47e7-94b2-d212002a7dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:06:03.102750 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 771/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 31 01:06:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:06:03.386644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d331fa-4d22-4f4e-a37e-3b31fe4d001a None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:06:03.403097 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d331fa-4d22-4f4e-a37e-3b31fe4d001a None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:06:03.408882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d659ea5f-6298-4c2f-9476-a3f8b6aff7cb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:06:03.409547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d659ea5f-6298-4c2f-9476-a3f8b6aff7cb service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:03.409829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d659ea5f-6298-4c2f-9476-a3f8b6aff7cb service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:03.410102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d659ea5f-6298-4c2f-9476-a3f8b6aff7cb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:03.420490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d659ea5f-6298-4c2f-9476-a3f8b6aff7cb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:06:03.422107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d659ea5f-6298-4c2f-9476-a3f8b6aff7cb service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:06:03.432907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 753/1524] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:06:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:06:04.311327 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a2ed2c-2c9b-4387-8834-fdfc7735cb5d None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:04.312226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a2ed2c-2c9b-4387-8834-fdfc7735cb5d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:04.312373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a2ed2c-2c9b-4387-8834-fdfc7735cb5d None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:04.314079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 772/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 31 01:06:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 01:06:04.320346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a00fdc-7c08-451d-80f9-a6ca27b65787 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:04.320937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a00fdc-7c08-451d-80f9-a6ca27b65787 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:04.321049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a00fdc-7c08-451d-80f9-a6ca27b65787 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:04.343188 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-23a00fdc-7c08-451d-80f9-a6ca27b65787 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:06:04.344461 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-23a00fdc-7c08-451d-80f9-a6ca27b65787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:06:04.353157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 754/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 31 01:06:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:06:04.508705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a2ed2c-2c9b-4387-8834-fdfc7735cb5d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:06:04.525270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a2ed2c-2c9b-4387-8834-fdfc7735cb5d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:06:04.531265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868fd05e-65e5-4f90-b4d1-c188bd08832d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:06:04.531922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868fd05e-65e5-4f90-b4d1-c188bd08832d service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:04.532202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868fd05e-65e5-4f90-b4d1-c188bd08832d service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:04.532474 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868fd05e-65e5-4f90-b4d1-c188bd08832d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:04.541873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868fd05e-65e5-4f90-b4d1-c188bd08832d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:06:04.543704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868fd05e-65e5-4f90-b4d1-c188bd08832d service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:06:04.555320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 773/1527] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:06:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:06:08.852244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8a9e3d-72a0-4ec4-a29f-f6d0c53d706c None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:08.852983 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8a9e3d-72a0-4ec4-a29f-f6d0c53d706c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:08.853179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8a9e3d-72a0-4ec4-a29f-f6d0c53d706c None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:08.855837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 755/1528] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:06:08 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:06:08.862036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0bad06-d084-4307-86ef-d24d446d66ef None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:08.862649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0bad06-d084-4307-86ef-d24d446d66ef None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:08.862756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0bad06-d084-4307-86ef-d24d446d66ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:08.884770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-6a0bad06-d084-4307-86ef-d24d446d66ef None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:06:08.886076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6a0bad06-d084-4307-86ef-d24d446d66ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:06:08.894476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 774/1529] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:06:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:06:09.020343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8a9e3d-72a0-4ec4-a29f-f6d0c53d706c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:06:09.024167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0d8a9e3d-72a0-4ec4-a29f-f6d0c53d706c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:06:09.174287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8a9e3d-72a0-4ec4-a29f-f6d0c53d706c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:06:09.180816 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c68e3b-a0ac-4bfa-b338-385e8f14a740 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:06:09.181896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c68e3b-a0ac-4bfa-b338-385e8f14a740 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:09.182375 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c68e3b-a0ac-4bfa-b338-385e8f14a740 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:09.182846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c68e3b-a0ac-4bfa-b338-385e8f14a740 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:09.200451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c68e3b-a0ac-4bfa-b338-385e8f14a740 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:06:09.203198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c68e3b-a0ac-4bfa-b338-385e8f14a740 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:06:09.220883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 756/1530] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:06:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:06:11.367601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b98d78a-a818-4985-8852-8e04ad982288 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:11.368235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b98d78a-a818-4985-8852-8e04ad982288 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:11.368504 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b98d78a-a818-4985-8852-8e04ad982288 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:11.370526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 775/1531] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:06:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 01:06:11.378702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff9c3d6-3c8a-4c12-8f3e-d756b48bfc7b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:11.379257 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff9c3d6-3c8a-4c12-8f3e-d756b48bfc7b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:11.379589 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff9c3d6-3c8a-4c12-8f3e-d756b48bfc7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:11.403835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-aff9c3d6-3c8a-4c12-8f3e-d756b48bfc7b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:06:11.412811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-aff9c3d6-3c8a-4c12-8f3e-d756b48bfc7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:06:11.423435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 757/1532] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:06:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:06:11.463139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b98d78a-a818-4985-8852-8e04ad982288 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:06:11.482512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b98d78a-a818-4985-8852-8e04ad982288 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:06:11.488961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679b51d2-b52a-4400-8f3e-30a6dd275185 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:06:11.490031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679b51d2-b52a-4400-8f3e-30a6dd275185 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:11.491059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679b51d2-b52a-4400-8f3e-30a6dd275185 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:11.491661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679b51d2-b52a-4400-8f3e-30a6dd275185 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:11.508849 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679b51d2-b52a-4400-8f3e-30a6dd275185 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:06:11.511686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679b51d2-b52a-4400-8f3e-30a6dd275185 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:06:11.527866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 776/1533] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:06:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:06:15.241727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234e1f8-ca78-4de2-86cf-972857aacff1 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:15.242380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234e1f8-ca78-4de2-86cf-972857aacff1 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:15.242380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234e1f8-ca78-4de2-86cf-972857aacff1 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:15.244495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 758/1534] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:06:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:06:15.251285 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda17764-5645-4cb1-b0f9-1c3db4f20d1e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:15.252487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda17764-5645-4cb1-b0f9-1c3db4f20d1e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:15.252487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda17764-5645-4cb1-b0f9-1c3db4f20d1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:15.275089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-eda17764-5645-4cb1-b0f9-1c3db4f20d1e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:06:15.276482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-eda17764-5645-4cb1-b0f9-1c3db4f20d1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:06:15.286107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 777/1535] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:06:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:06:15.631978 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f234e1f8-ca78-4de2-86cf-972857aacff1 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:06:15.634815 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f234e1f8-ca78-4de2-86cf-972857aacff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:06:15.738265 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:06:15.756021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f234e1f8-ca78-4de2-86cf-972857aacff1 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:06:15.760247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f730c5-f648-4a26-91f3-5900be48096d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:06:15.760973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f730c5-f648-4a26-91f3-5900be48096d service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:15.761294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f730c5-f648-4a26-91f3-5900be48096d service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:15.761636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f730c5-f648-4a26-91f3-5900be48096d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:15.772576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f730c5-f648-4a26-91f3-5900be48096d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:06:15.774050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f730c5-f648-4a26-91f3-5900be48096d service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:06:15.785452 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 759/1536] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:06:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:06:18.124753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcf6c47-1ed2-4480-aec0-d26f1aac17b4 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:18.125414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcf6c47-1ed2-4480-aec0-d26f1aac17b4 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:18.125414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcf6c47-1ed2-4480-aec0-d26f1aac17b4 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:18.127292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 778/1537] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 31 01:06:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 01:06:18.135771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45505e6-2f04-47f6-8c55-4917210a0561 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:18.136373 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45505e6-2f04-47f6-8c55-4917210a0561 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:18.136651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45505e6-2f04-47f6-8c55-4917210a0561 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:18.171428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b45505e6-2f04-47f6-8c55-4917210a0561 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:06:18.173666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b45505e6-2f04-47f6-8c55-4917210a0561 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:06:18.188310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 760/1538] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 31 01:06:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:06:18.533389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcf6c47-1ed2-4480-aec0-d26f1aac17b4 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:06:18.536197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-bfcf6c47-1ed2-4480-aec0-d26f1aac17b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:06:18.549611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=84 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:06:18.558184 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcf6c47-1ed2-4480-aec0-d26f1aac17b4 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:06:18.562308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3458c1b6-34d9-4c20-9197-9c1327a31b70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:06:18.562996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3458c1b6-34d9-4c20-9197-9c1327a31b70 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:06:18.563273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3458c1b6-34d9-4c20-9197-9c1327a31b70 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:06:18.563570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3458c1b6-34d9-4c20-9197-9c1327a31b70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:06:18.575754 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3458c1b6-34d9-4c20-9197-9c1327a31b70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:06:18.577222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3458c1b6-34d9-4c20-9197-9c1327a31b70 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:06:18.589934 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 779/1539] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:06:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:06:28.163150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=11 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:06:28.584817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:07:55.654279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0c75d0-af84-40fb-969f-9d0150f84ad0 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:55.659483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0c75d0-af84-40fb-969f-9d0150f84ad0 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:55.659483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0c75d0-af84-40fb-969f-9d0150f84ad0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:55.709527 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-fa0c75d0-af84-40fb-969f-9d0150f84ad0 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:55.716649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fa0c75d0-af84-40fb-969f-9d0150f84ad0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:55.745717 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 761/1540] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:55.768521 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f514e3c0-0d6e-4c8d-92b1-5f8008c37f8c None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:55.770027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f514e3c0-0d6e-4c8d-92b1-5f8008c37f8c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:55.770027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f514e3c0-0d6e-4c8d-92b1-5f8008c37f8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:55.821455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f514e3c0-0d6e-4c8d-92b1-5f8008c37f8c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:55.823869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f514e3c0-0d6e-4c8d-92b1-5f8008c37f8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:55.845031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 780/1541] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:07:55.852933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0c75d0-af84-40fb-969f-9d0150f84ad0 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:55.876427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0c75d0-af84-40fb-969f-9d0150f84ad0 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:55.891823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d5eeac-0084-4bb3-9cf3-2df69721f950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:55.892904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d5eeac-0084-4bb3-9cf3-2df69721f950 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:55.893397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d5eeac-0084-4bb3-9cf3-2df69721f950 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:55.893923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d5eeac-0084-4bb3-9cf3-2df69721f950 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:55.895176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d5eeac-0084-4bb3-9cf3-2df69721f950 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:07:55.895766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d5eeac-0084-4bb3-9cf3-2df69721f950 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:55.899441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d5eeac-0084-4bb3-9cf3-2df69721f950 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:55.933853 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 762/1542] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:07:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:07:55.965891 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f514e3c0-0d6e-4c8d-92b1-5f8008c37f8c None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:55.990003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f514e3c0-0d6e-4c8d-92b1-5f8008c37f8c None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:56.000990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3372874-4550-41d1-a65f-df3645f6873a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:56.002046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3372874-4550-41d1-a65f-df3645f6873a admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:56.002534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3372874-4550-41d1-a65f-df3645f6873a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:56.003002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3372874-4550-41d1-a65f-df3645f6873a admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:56.004935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3372874-4550-41d1-a65f-df3645f6873a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:56.008636 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3372874-4550-41d1-a65f-df3645f6873a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:56.125470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 781/1543] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:07:55 2022] POST /identity/v3/projects => generated 367 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:07:56.132508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d5eeac-0084-4bb3-9cf3-2df69721f950 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:56.149294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d5eeac-0084-4bb3-9cf3-2df69721f950 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:56.178569 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9923234d-18f2-4dc6-9a39-d56d7bf0975d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:56.179047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9923234d-18f2-4dc6-9a39-d56d7bf0975d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:56.179542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9923234d-18f2-4dc6-9a39-d56d7bf0975d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:56.246671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-9923234d-18f2-4dc6-9a39-d56d7bf0975d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:56.248927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9923234d-18f2-4dc6-9a39-d56d7bf0975d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:56.305877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003a039c-1a97-4019-b009-66a285994504 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:56.306524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003a039c-1a97-4019-b009-66a285994504 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:56.306816 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003a039c-1a97-4019-b009-66a285994504 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:56.307135 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003a039c-1a97-4019-b009-66a285994504 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:56.307791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003a039c-1a97-4019-b009-66a285994504 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:56.310072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003a039c-1a97-4019-b009-66a285994504 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:56.372601 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 763/1544] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:07:56 2022] POST /identity/v3/users => generated 284 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:07:56.384986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003a039c-1a97-4019-b009-66a285994504 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:56.405794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003a039c-1a97-4019-b009-66a285994504 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:56.421746 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea2da99-436f-46d9-bd92-741c94a7f0c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:56.422399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea2da99-436f-46d9-bd92-741c94a7f0c2 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:56.424737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea2da99-436f-46d9-bd92-741c94a7f0c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:56.425052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea2da99-436f-46d9-bd92-741c94a7f0c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:56.426321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea2da99-436f-46d9-bd92-741c94a7f0c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:56.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea2da99-436f-46d9-bd92-741c94a7f0c2 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:56.454408 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 782/1545] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:56.455803 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 764/1546] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:56 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:56.465108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9923234d-18f2-4dc6-9a39-d56d7bf0975d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:56.473393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ab2d7a-fb9c-47b9-b9fa-a0ba24bf4fc5 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:56.473697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ab2d7a-fb9c-47b9-b9fa-a0ba24bf4fc5 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:56.474021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ab2d7a-fb9c-47b9-b9fa-a0ba24bf4fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:56.485562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9923234d-18f2-4dc6-9a39-d56d7bf0975d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:56.504503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3c63b6-fae0-44e8-aa58-753fb27f1b48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:56.505161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3c63b6-fae0-44e8-aa58-753fb27f1b48 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:56.505449 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3c63b6-fae0-44e8-aa58-753fb27f1b48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:56.505764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3c63b6-fae0-44e8-aa58-753fb27f1b48 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:56.506456 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3c63b6-fae0-44e8-aa58-753fb27f1b48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:56.508312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3c63b6-fae0-44e8-aa58-753fb27f1b48 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:56.525731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-81ab2d7a-fb9c-47b9-b9fa-a0ba24bf4fc5 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:56.527046 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-81ab2d7a-fb9c-47b9-b9fa-a0ba24bf4fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:56.533432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 783/1547] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:56 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:56.540522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3c63b6-fae0-44e8-aa58-753fb27f1b48 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:56.554244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 765/1548] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:07:56.564398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ab2d7a-fb9c-47b9-b9fa-a0ba24bf4fc5 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:56.567796 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3c63b6-fae0-44e8-aa58-753fb27f1b48 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:56.596868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd4d77a-8d14-40fe-9093-cc7650b74f03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:56.598255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd4d77a-8d14-40fe-9093-cc7650b74f03 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:56.598687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd4d77a-8d14-40fe-9093-cc7650b74f03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:56.599108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd4d77a-8d14-40fe-9093-cc7650b74f03 admin admin] PATH_INFO: `/v3/projects/e29bf692ac1845bbac73e28693875517/users/cf8d97711043403b95da4c0d1fc6942f/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:56.610229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ab2d7a-fb9c-47b9-b9fa-a0ba24bf4fc5 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:56.630036 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ccdcb3-b579-482d-ab17-0de1ce3cb708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:56.630728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ccdcb3-b579-482d-ab17-0de1ce3cb708 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:56.631015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ccdcb3-b579-482d-ab17-0de1ce3cb708 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:56.631292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ccdcb3-b579-482d-ab17-0de1ce3cb708 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:56.632065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ccdcb3-b579-482d-ab17-0de1ce3cb708 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:07:56.632376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ccdcb3-b579-482d-ab17-0de1ce3cb708 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:56.634074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ccdcb3-b579-482d-ab17-0de1ce3cb708 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:56.667953 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 766/1549] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:07:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:07:56.682277 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd4d77a-8d14-40fe-9093-cc7650b74f03 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e29bf692ac1845bbac73e28693875517, user_id=cf8d97711043403b95da4c0d1fc6942f, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:56.688017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd4d77a-8d14-40fe-9093-cc7650b74f03 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:56.696536 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ccdcb3-b579-482d-ab17-0de1ce3cb708 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:56.717159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ccdcb3-b579-482d-ab17-0de1ce3cb708 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:56.751473 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 784/1550] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:07:56 2022] PUT /identity/v3/projects/e29bf692ac1845bbac73e28693875517/users/cf8d97711043403b95da4c0d1fc6942f/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:07:56.791584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ef8cac-64d6-44a1-84aa-c92c3e957de8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:56.792288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ef8cac-64d6-44a1-84aa-c92c3e957de8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:56.792544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ef8cac-64d6-44a1-84aa-c92c3e957de8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:56.837716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcceb7f7-a48c-4483-a8e9-6584bf2895a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:56.838377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcceb7f7-a48c-4483-a8e9-6584bf2895a1 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:56.838664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcceb7f7-a48c-4483-a8e9-6584bf2895a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:56.838944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcceb7f7-a48c-4483-a8e9-6584bf2895a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:56.839658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcceb7f7-a48c-4483-a8e9-6584bf2895a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:56.840711 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: WARNING py.warnings [None req-dcceb7f7-a48c-4483-a8e9-6584bf2895a1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:07:56.840711 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: warnings.warn(msg) Jul 31 01:07:56.840711 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: Jul 31 01:07:56.841950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcceb7f7-a48c-4483-a8e9-6584bf2895a1 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:56.893186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 767/1551] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:07:56 2022] POST /identity/v3/projects => generated 379 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:56.898041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-e9ef8cac-64d6-44a1-84aa-c92c3e957de8 None None] MFA Rules not processed for user `cf8d97711043403b95da4c0d1fc6942f`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:56.900648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e9ef8cac-64d6-44a1-84aa-c92c3e957de8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:56.900832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcceb7f7-a48c-4483-a8e9-6584bf2895a1 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:56.944186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:07:57.026460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcceb7f7-a48c-4483-a8e9-6584bf2895a1 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:57.045875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd58225e-1386-44e7-82a5-2633d46819d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:57.047344 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd58225e-1386-44e7-82a5-2633d46819d9 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.048174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd58225e-1386-44e7-82a5-2633d46819d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.048908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd58225e-1386-44e7-82a5-2633d46819d9 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.050958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd58225e-1386-44e7-82a5-2633d46819d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:57.059280 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd58225e-1386-44e7-82a5-2633d46819d9 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:57.099426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 785/1552] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:56 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 310 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:57.101136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9953f671-dc6e-48f4-9f61-30c096bb2ee5 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.101496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9953f671-dc6e-48f4-9f61-30c096bb2ee5 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.101892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9953f671-dc6e-48f4-9f61-30c096bb2ee5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.118724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 768/1553] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:07:56 2022] POST /identity/v3/users => generated 290 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:07:57.120646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd58225e-1386-44e7-82a5-2633d46819d9 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:57.152678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-9953f671-dc6e-48f4-9f61-30c096bb2ee5 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:57.154043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9953f671-dc6e-48f4-9f61-30c096bb2ee5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:57.154855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd58225e-1386-44e7-82a5-2633d46819d9 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:57.169123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 786/1554] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:57.169706 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9953f671-dc6e-48f4-9f61-30c096bb2ee5 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:57.172100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941a8be7-c9c2-413d-b9af-8d6d620c61b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:57.173536 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941a8be7-c9c2-413d-b9af-8d6d620c61b8 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.174147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941a8be7-c9c2-413d-b9af-8d6d620c61b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.174719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941a8be7-c9c2-413d-b9af-8d6d620c61b8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.181009 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941a8be7-c9c2-413d-b9af-8d6d620c61b8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:57.187163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941a8be7-c9c2-413d-b9af-8d6d620c61b8 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:57.191956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9953f671-dc6e-48f4-9f61-30c096bb2ee5 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:57.212297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a2d344-c801-4527-a300-951198aa6847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:57.212868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a2d344-c801-4527-a300-951198aa6847 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.213152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a2d344-c801-4527-a300-951198aa6847 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.213429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a2d344-c801-4527-a300-951198aa6847 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a2d344-c801-4527-a300-951198aa6847 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:57.215993 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a2d344-c801-4527-a300-951198aa6847 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:57.229104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:07:57.247930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 787/1555] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:57 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:57.249378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c733465-537c-4d4b-b1b4-a4400dd23568 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.249655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c733465-537c-4d4b-b1b4-a4400dd23568 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.249936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c733465-537c-4d4b-b1b4-a4400dd23568 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.274854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 769/1556] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/projects => generated 365 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:57.275788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941a8be7-c9c2-413d-b9af-8d6d620c61b8 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:57.302672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-4c733465-537c-4d4b-b1b4-a4400dd23568 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:57.304917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4c733465-537c-4d4b-b1b4-a4400dd23568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:57.459287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941a8be7-c9c2-413d-b9af-8d6d620c61b8 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:57.477069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9295f3-6f69-43d5-b786-550b416bdfcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:57.477633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9295f3-6f69-43d5-b786-550b416bdfcd admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.477954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9295f3-6f69-43d5-b786-550b416bdfcd admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.478249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9295f3-6f69-43d5-b786-550b416bdfcd admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.479180 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9295f3-6f69-43d5-b786-550b416bdfcd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:57.480921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9295f3-6f69-43d5-b786-550b416bdfcd admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:57.524560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 770/1557] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:57 2022] GET /identity/v3/roles => generated 1420 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:57.526995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ba65d3-1534-4143-9fb0-8f709e7cb468 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.527592 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ba65d3-1534-4143-9fb0-8f709e7cb468 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.530475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ba65d3-1534-4143-9fb0-8f709e7cb468 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.534567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 788/1558] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:57.535678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c733465-537c-4d4b-b1b4-a4400dd23568 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:57.570906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c733465-537c-4d4b-b1b4-a4400dd23568 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:57.587227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aeec057-a37d-4a60-9ff1-d2f972c38f84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:57.589892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeec057-a37d-4a60-9ff1-d2f972c38f84 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.590477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeec057-a37d-4a60-9ff1-d2f972c38f84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.590951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeec057-a37d-4a60-9ff1-d2f972c38f84 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.595943 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aeec057-a37d-4a60-9ff1-d2f972c38f84 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:57.597980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: WARNING py.warnings [None req-3aeec057-a37d-4a60-9ff1-d2f972c38f84 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:07:57.597980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: warnings.warn(msg) Jul 31 01:07:57.597980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: Jul 31 01:07:57.601390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a8ba65d3-1534-4143-9fb0-8f709e7cb468 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:57.603594 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a8ba65d3-1534-4143-9fb0-8f709e7cb468 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:57.606452 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aeec057-a37d-4a60-9ff1-d2f972c38f84 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:57.621077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 771/1559] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:57.629229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1ac69-e624-43e6-9aff-42e4db009378 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.629543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1ac69-e624-43e6-9aff-42e4db009378 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.633697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1ac69-e624-43e6-9aff-42e4db009378 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.654886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 789/1560] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/users => generated 289 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:57.657827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aeec057-a37d-4a60-9ff1-d2f972c38f84 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:57.670392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-32c1ac69-e624-43e6-9aff-42e4db009378 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:57.672495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-32c1ac69-e624-43e6-9aff-42e4db009378 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:57.676821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aeec057-a37d-4a60-9ff1-d2f972c38f84 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:57.685168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a2a27b-9da3-4c34-b2f6-1a2e619dcc25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:57.686603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a2a27b-9da3-4c34-b2f6-1a2e619dcc25 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.687103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a2a27b-9da3-4c34-b2f6-1a2e619dcc25 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.687632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a2a27b-9da3-4c34-b2f6-1a2e619dcc25 admin admin] PATH_INFO: `/v3/projects/66c7f61afd0d4650bd530eff18a2318b/users/c4a002047e3e42338e4f57ba49e877fb/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.697598 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 772/1561] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:57.709186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c1ac69-e624-43e6-9aff-42e4db009378 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:57.719536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a2a27b-9da3-4c34-b2f6-1a2e619dcc25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=66c7f61afd0d4650bd530eff18a2318b, user_id=c4a002047e3e42338e4f57ba49e877fb, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:57.725319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a2a27b-9da3-4c34-b2f6-1a2e619dcc25 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:57.725964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c1ac69-e624-43e6-9aff-42e4db009378 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:57.733152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57268c16-e294-46ce-9a09-53f7f2134cbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:57.733877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57268c16-e294-46ce-9a09-53f7f2134cbf admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.734341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57268c16-e294-46ce-9a09-53f7f2134cbf admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.734777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57268c16-e294-46ce-9a09-53f7f2134cbf admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.736058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57268c16-e294-46ce-9a09-53f7f2134cbf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:07:57.736550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57268c16-e294-46ce-9a09-53f7f2134cbf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:57.739994 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57268c16-e294-46ce-9a09-53f7f2134cbf admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:57.755556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 790/1562] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:07:57 2022] PUT /identity/v3/projects/66c7f61afd0d4650bd530eff18a2318b/users/c4a002047e3e42338e4f57ba49e877fb/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:07:57.755556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7acdbd3-3add-4eda-8011-eee50b0b91a8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.755556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7acdbd3-3add-4eda-8011-eee50b0b91a8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.755556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7acdbd3-3add-4eda-8011-eee50b0b91a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.760796 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 773/1563] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:07:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:07:57.762826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ee4ca3-fb05-400f-83ed-652f674c03a9 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.763299 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ee4ca3-fb05-400f-83ed-652f674c03a9 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.763777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ee4ca3-fb05-400f-83ed-652f674c03a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.789590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-e7acdbd3-3add-4eda-8011-eee50b0b91a8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:57.791544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e7acdbd3-3add-4eda-8011-eee50b0b91a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:57.795092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d3ee4ca3-fb05-400f-83ed-652f674c03a9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:57.797091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d3ee4ca3-fb05-400f-83ed-652f674c03a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:57.876096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 791/1564] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:57.879470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc27200-98d8-4349-981e-0d01d5b88b25 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.880062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc27200-98d8-4349-981e-0d01d5b88b25 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.880547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc27200-98d8-4349-981e-0d01d5b88b25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.888326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 774/1565] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:57.889409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ee4ca3-fb05-400f-83ed-652f674c03a9 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:57.897260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:07:57.909979 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ee4ca3-fb05-400f-83ed-652f674c03a9 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:57.910434 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-adc27200-98d8-4349-981e-0d01d5b88b25 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:57.912102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-adc27200-98d8-4349-981e-0d01d5b88b25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:57.917839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ab960a-1f6f-4a69-908d-baae197b6010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:57.918675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ab960a-1f6f-4a69-908d-baae197b6010 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.919067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ab960a-1f6f-4a69-908d-baae197b6010 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.919489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ab960a-1f6f-4a69-908d-baae197b6010 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.922655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ab960a-1f6f-4a69-908d-baae197b6010 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:57.922880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 792/1566] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:57.923738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ab960a-1f6f-4a69-908d-baae197b6010 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:57.923957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab473f5c-c77c-4575-a2e0-07e770f5dbd8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.924236 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab473f5c-c77c-4575-a2e0-07e770f5dbd8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.924505 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab473f5c-c77c-4575-a2e0-07e770f5dbd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.943076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 775/1567] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:57 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:57.944541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684d1b15-9e00-4738-be37-f9f0faf5717c None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.944879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684d1b15-9e00-4738-be37-f9f0faf5717c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.945229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684d1b15-9e00-4738-be37-f9f0faf5717c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.957894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ab473f5c-c77c-4575-a2e0-07e770f5dbd8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:57.960045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ab473f5c-c77c-4575-a2e0-07e770f5dbd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:57.966512 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-684d1b15-9e00-4738-be37-f9f0faf5717c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:57.968128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-684d1b15-9e00-4738-be37-f9f0faf5717c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:57.974860 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 793/1568] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:57.977560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e444cd-1629-400d-a239-c16d3296f457 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:57.977968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e444cd-1629-400d-a239-c16d3296f457 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:57.978417 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e444cd-1629-400d-a239-c16d3296f457 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:57.981509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 776/1569] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:57.983021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684d1b15-9e00-4738-be37-f9f0faf5717c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.001463 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684d1b15-9e00-4738-be37-f9f0faf5717c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.006471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-15e444cd-1629-400d-a239-c16d3296f457 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:58.007730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-15e444cd-1629-400d-a239-c16d3296f457 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:58.009624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db764de6-1e7f-40a0-a924-739f8ca4ba62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.010723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db764de6-1e7f-40a0-a924-739f8ca4ba62 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.011251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db764de6-1e7f-40a0-a924-739f8ca4ba62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.011818 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db764de6-1e7f-40a0-a924-739f8ca4ba62 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.013066 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db764de6-1e7f-40a0-a924-739f8ca4ba62 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.016100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 794/1570] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:58.017008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db764de6-1e7f-40a0-a924-739f8ca4ba62 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.017418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d223bc-7281-4485-b461-86b03836aa3e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.017781 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d223bc-7281-4485-b461-86b03836aa3e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.018097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d223bc-7281-4485-b461-86b03836aa3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.041574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-34d223bc-7281-4485-b461-86b03836aa3e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:58.042801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-34d223bc-7281-4485-b461-86b03836aa3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:58.052315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 795/1571] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:58.053979 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d223bc-7281-4485-b461-86b03836aa3e None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.072324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 777/1572] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:07:57 2022] POST /identity/v3/projects => generated 385 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:58.072610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d223bc-7281-4485-b461-86b03836aa3e None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.072748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa30cbc-98f3-4926-abee-2940e1204e26 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.072748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa30cbc-98f3-4926-abee-2940e1204e26 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.073037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa30cbc-98f3-4926-abee-2940e1204e26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.105300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-daa30cbc-98f3-4926-abee-2940e1204e26 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:58.107229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-daa30cbc-98f3-4926-abee-2940e1204e26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:58.123392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:07:58.157334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff58bfdb-44b7-4a10-8670-3dbe8351e442 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.157971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff58bfdb-44b7-4a10-8670-3dbe8351e442 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.158252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff58bfdb-44b7-4a10-8670-3dbe8351e442 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.158529 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff58bfdb-44b7-4a10-8670-3dbe8351e442 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.159329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff58bfdb-44b7-4a10-8670-3dbe8351e442 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:07:58.160176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff58bfdb-44b7-4a10-8670-3dbe8351e442 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.161219 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: WARNING py.warnings [None req-ff58bfdb-44b7-4a10-8670-3dbe8351e442 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:07:58.161219 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: warnings.warn(msg) Jul 31 01:07:58.161219 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: Jul 31 01:07:58.162139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff58bfdb-44b7-4a10-8670-3dbe8351e442 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.187394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 796/1573] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:07:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:07:58.187394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff58bfdb-44b7-4a10-8670-3dbe8351e442 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.188021 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff58bfdb-44b7-4a10-8670-3dbe8351e442 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.196140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ba22c0-dabe-4f64-b3ea-94e0795e87bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.197034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ba22c0-dabe-4f64-b3ea-94e0795e87bf admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.197257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ba22c0-dabe-4f64-b3ea-94e0795e87bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.197733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ba22c0-dabe-4f64-b3ea-94e0795e87bf admin admin] PATH_INFO: `/v3/projects/9fb1fc3901ef47cc81c384173376f0d5/users/d8eae4b7123a445db9a3abace47ea29b/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.216546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 778/1574] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:58.217522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa30cbc-98f3-4926-abee-2940e1204e26 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.222977 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ba22c0-dabe-4f64-b3ea-94e0795e87bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fb1fc3901ef47cc81c384173376f0d5, user_id=d8eae4b7123a445db9a3abace47ea29b, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.225836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ba22c0-dabe-4f64-b3ea-94e0795e87bf admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.233044 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa30cbc-98f3-4926-abee-2940e1204e26 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.241530 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 797/1575] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:07:58 2022] PUT /identity/v3/projects/9fb1fc3901ef47cc81c384173376f0d5/users/d8eae4b7123a445db9a3abace47ea29b/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:07:58.242924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcac55f6-b3bb-484c-a9ac-815dd280c348 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.243316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcac55f6-b3bb-484c-a9ac-815dd280c348 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.243745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcac55f6-b3bb-484c-a9ac-815dd280c348 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.265956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-bcac55f6-b3bb-484c-a9ac-815dd280c348 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:58.267090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-bcac55f6-b3bb-484c-a9ac-815dd280c348 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:58.326703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e682b58-a7e9-4e16-892c-0a8665277c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.327481 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e682b58-a7e9-4e16-892c-0a8665277c9b admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.327766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e682b58-a7e9-4e16-892c-0a8665277c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.328043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e682b58-a7e9-4e16-892c-0a8665277c9b admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.328757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e682b58-a7e9-4e16-892c-0a8665277c9b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:07:58.329510 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e682b58-a7e9-4e16-892c-0a8665277c9b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.330712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e682b58-a7e9-4e16-892c-0a8665277c9b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.346364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 779/1576] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:07:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:07:58.348379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e682b58-a7e9-4e16-892c-0a8665277c9b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.351472 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 798/1577] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:58.352246 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcac55f6-b3bb-484c-a9ac-815dd280c348 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.362503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcac55f6-b3bb-484c-a9ac-815dd280c348 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.367551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d096c3-b9c5-4b96-a037-215dd433110d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.368179 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d096c3-b9c5-4b96-a037-215dd433110d admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.368469 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d096c3-b9c5-4b96-a037-215dd433110d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.368751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d096c3-b9c5-4b96-a037-215dd433110d admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.369465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d096c3-b9c5-4b96-a037-215dd433110d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:07:58.369772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d096c3-b9c5-4b96-a037-215dd433110d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.371479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d096c3-b9c5-4b96-a037-215dd433110d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.388184 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 799/1578] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:07:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:07:58.388184 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d096c3-b9c5-4b96-a037-215dd433110d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.403162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d096c3-b9c5-4b96-a037-215dd433110d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.409214 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badd5b21-62a5-4a5e-8c82-c515d10de07d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.410451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badd5b21-62a5-4a5e-8c82-c515d10de07d admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.410781 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badd5b21-62a5-4a5e-8c82-c515d10de07d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.411123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badd5b21-62a5-4a5e-8c82-c515d10de07d admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.411896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badd5b21-62a5-4a5e-8c82-c515d10de07d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.413973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badd5b21-62a5-4a5e-8c82-c515d10de07d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.434602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e682b58-a7e9-4e16-892c-0a8665277c9b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.440314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 800/1579] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:07:58 2022] POST /identity/v3/users => generated 293 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:58.440314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badd5b21-62a5-4a5e-8c82-c515d10de07d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.443376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a478971c-1453-4cee-82bc-2f739004ef0f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.444210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a478971c-1453-4cee-82bc-2f739004ef0f service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.444606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a478971c-1453-4cee-82bc-2f739004ef0f service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.445006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a478971c-1453-4cee-82bc-2f739004ef0f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.449477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badd5b21-62a5-4a5e-8c82-c515d10de07d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.454578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80539ad-d26b-4a50-8cf4-7b1b67882fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.455210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80539ad-d26b-4a50-8cf4-7b1b67882fb2 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.455549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80539ad-d26b-4a50-8cf4-7b1b67882fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.455826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80539ad-d26b-4a50-8cf4-7b1b67882fb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.457090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80539ad-d26b-4a50-8cf4-7b1b67882fb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.457799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a478971c-1453-4cee-82bc-2f739004ef0f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.459010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80539ad-d26b-4a50-8cf4-7b1b67882fb2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.459548 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a478971c-1453-4cee-82bc-2f739004ef0f service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.465761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:07:58.475588 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 780/1580] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:07:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:07:58.478444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a478971c-1453-4cee-82bc-2f739004ef0f service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.490261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 801/1581] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:07:58 2022] POST /identity/v3/projects => generated 375 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:58.493712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a478971c-1453-4cee-82bc-2f739004ef0f service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.494887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80539ad-d26b-4a50-8cf4-7b1b67882fb2 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.567267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:07:58.582355 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80539ad-d26b-4a50-8cf4-7b1b67882fb2 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.591737 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5737f26-5715-4bb1-ba84-e6b19e7334ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.592398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5737f26-5715-4bb1-ba84-e6b19e7334ec admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.592730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5737f26-5715-4bb1-ba84-e6b19e7334ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.593013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5737f26-5715-4bb1-ba84-e6b19e7334ec admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.594440 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5737f26-5715-4bb1-ba84-e6b19e7334ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.596192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5737f26-5715-4bb1-ba84-e6b19e7334ec admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.599123 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae44a4e4-36b8-4ec2-92c7-8cc924ea0df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.600181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae44a4e4-36b8-4ec2-92c7-8cc924ea0df7 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.600677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae44a4e4-36b8-4ec2-92c7-8cc924ea0df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.601152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae44a4e4-36b8-4ec2-92c7-8cc924ea0df7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.602889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae44a4e4-36b8-4ec2-92c7-8cc924ea0df7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:07:58.603460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae44a4e4-36b8-4ec2-92c7-8cc924ea0df7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.606550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae44a4e4-36b8-4ec2-92c7-8cc924ea0df7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.611641 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 802/1582] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:58 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:58.612889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5737f26-5715-4bb1-ba84-e6b19e7334ec admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.629705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5737f26-5715-4bb1-ba84-e6b19e7334ec admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.630060 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 781/1583] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:07:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:07:58.630936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae44a4e4-36b8-4ec2-92c7-8cc924ea0df7 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.641701 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae44a4e4-36b8-4ec2-92c7-8cc924ea0df7 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.648636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad12a01b-c76b-4a1e-9243-d09f091f1e8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.649495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad12a01b-c76b-4a1e-9243-d09f091f1e8c admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.649754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad12a01b-c76b-4a1e-9243-d09f091f1e8c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.650030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad12a01b-c76b-4a1e-9243-d09f091f1e8c admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.650659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5255b9ef-40b8-4ef7-b7e6-cc1047282cff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.650833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad12a01b-c76b-4a1e-9243-d09f091f1e8c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:07:58.651088 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad12a01b-c76b-4a1e-9243-d09f091f1e8c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.651199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5255b9ef-40b8-4ef7-b7e6-cc1047282cff admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.651622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5255b9ef-40b8-4ef7-b7e6-cc1047282cff admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.651957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5255b9ef-40b8-4ef7-b7e6-cc1047282cff admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.652649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5255b9ef-40b8-4ef7-b7e6-cc1047282cff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.653302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad12a01b-c76b-4a1e-9243-d09f091f1e8c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.654798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5255b9ef-40b8-4ef7-b7e6-cc1047282cff admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.677977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 782/1584] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:07:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:07:58.677977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad12a01b-c76b-4a1e-9243-d09f091f1e8c admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.691108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad12a01b-c76b-4a1e-9243-d09f091f1e8c admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.697811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9715f972-d99d-40eb-9d51-f699d2aefc00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.698453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9715f972-d99d-40eb-9d51-f699d2aefc00 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.698881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9715f972-d99d-40eb-9d51-f699d2aefc00 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.703315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 803/1585] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:07:58 2022] POST /identity/v3/projects => generated 381 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:58.703525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9715f972-d99d-40eb-9d51-f699d2aefc00 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.703641 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5255b9ef-40b8-4ef7-b7e6-cc1047282cff admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.703736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9715f972-d99d-40eb-9d51-f699d2aefc00 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.703736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9715f972-d99d-40eb-9d51-f699d2aefc00 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.743715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 783/1586] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:07:58 2022] POST /identity/v3/projects => generated 403 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:58.743715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9715f972-d99d-40eb-9d51-f699d2aefc00 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.788574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5255b9ef-40b8-4ef7-b7e6-cc1047282cff admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.837457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9715f972-d99d-40eb-9d51-f699d2aefc00 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.844494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2731ed-9afb-4612-aad3-a0873cebb41e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.845401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2731ed-9afb-4612-aad3-a0873cebb41e admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.845814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2731ed-9afb-4612-aad3-a0873cebb41e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.846164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2731ed-9afb-4612-aad3-a0873cebb41e admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.847012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2731ed-9afb-4612-aad3-a0873cebb41e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.849790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2731ed-9afb-4612-aad3-a0873cebb41e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.879624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 784/1587] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:07:58 2022] POST /identity/v3/users => generated 288 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:58.880496 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2731ed-9afb-4612-aad3-a0873cebb41e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.880901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4056b08-0de1-439f-9aa5-1a556c2247b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.881585 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4056b08-0de1-439f-9aa5-1a556c2247b1 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.881941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4056b08-0de1-439f-9aa5-1a556c2247b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.882311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4056b08-0de1-439f-9aa5-1a556c2247b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.883096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4056b08-0de1-439f-9aa5-1a556c2247b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.885098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4056b08-0de1-439f-9aa5-1a556c2247b1 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.892417 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2731ed-9afb-4612-aad3-a0873cebb41e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.897644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfc7608-d226-43d9-9563-d047c154ac35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:58.897894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 804/1588] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:58 2022] GET /identity/v3/roles => generated 1420 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:58.898362 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfc7608-d226-43d9-9563-d047c154ac35 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:58.898705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfc7608-d226-43d9-9563-d047c154ac35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:58.898832 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4056b08-0de1-439f-9aa5-1a556c2247b1 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.899205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfc7608-d226-43d9-9563-d047c154ac35 admin admin] PATH_INFO: `/v3/projects/9fb1fc3901ef47cc81c384173376f0d5/users/d8eae4b7123a445db9a3abace47ea29b/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:58.901325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfc7608-d226-43d9-9563-d047c154ac35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fb1fc3901ef47cc81c384173376f0d5, user_id=d8eae4b7123a445db9a3abace47ea29b, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:58.902468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: WARNING py.warnings [None req-ccfc7608-d226-43d9-9563-d047c154ac35 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:07:58.902468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: warnings.warn(msg) Jul 31 01:07:58.902468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: Jul 31 01:07:58.904493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfc7608-d226-43d9-9563-d047c154ac35 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:58.909840 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4056b08-0de1-439f-9aa5-1a556c2247b1 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:58.919090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 785/1589] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:07:58 2022] PUT /identity/v3/projects/9fb1fc3901ef47cc81c384173376f0d5/users/d8eae4b7123a445db9a3abace47ea29b/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:07:58.920838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfc7608-d226-43d9-9563-d047c154ac35 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:58.982184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:07:59.005006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d075e0-12de-4ba9-8dcd-e8cb8d4ee9d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.005418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d075e0-12de-4ba9-8dcd-e8cb8d4ee9d2 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.005563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d075e0-12de-4ba9-8dcd-e8cb8d4ee9d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.006094 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d075e0-12de-4ba9-8dcd-e8cb8d4ee9d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.006985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d075e0-12de-4ba9-8dcd-e8cb8d4ee9d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.008846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d075e0-12de-4ba9-8dcd-e8cb8d4ee9d2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.025988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:07:59.026721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfc7608-d226-43d9-9563-d047c154ac35 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.039016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7644c115-dd47-47c1-ac53-f1fdec3015a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.040026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7644c115-dd47-47c1-ac53-f1fdec3015a1 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.040553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7644c115-dd47-47c1-ac53-f1fdec3015a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.041008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7644c115-dd47-47c1-ac53-f1fdec3015a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.042259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7644c115-dd47-47c1-ac53-f1fdec3015a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.045865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7644c115-dd47-47c1-ac53-f1fdec3015a1 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.046715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 805/1590] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:07:58 2022] POST /identity/v3/projects => generated 389 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:59.047902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d075e0-12de-4ba9-8dcd-e8cb8d4ee9d2 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.098896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 786/1591] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:07:58 2022] POST /identity/v3/projects => generated 365 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:59.102868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7644c115-dd47-47c1-ac53-f1fdec3015a1 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.158630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d075e0-12de-4ba9-8dcd-e8cb8d4ee9d2 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.254179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7644c115-dd47-47c1-ac53-f1fdec3015a1 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.261972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cc14a5-813a-44b8-9784-75d297818c43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.263309 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cc14a5-813a-44b8-9784-75d297818c43 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.264219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cc14a5-813a-44b8-9784-75d297818c43 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.264760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cc14a5-813a-44b8-9784-75d297818c43 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.269376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cc14a5-813a-44b8-9784-75d297818c43 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.273335 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cc14a5-813a-44b8-9784-75d297818c43 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.298447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4866554-f59e-4d06-a33e-da556914b10a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.299872 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4866554-f59e-4d06-a33e-da556914b10a admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.300409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4866554-f59e-4d06-a33e-da556914b10a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.300858 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4866554-f59e-4d06-a33e-da556914b10a admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.302090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4866554-f59e-4d06-a33e-da556914b10a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.305869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4866554-f59e-4d06-a33e-da556914b10a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.306565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 787/1592] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:07:59 2022] POST /identity/v3/users => generated 302 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:59.308376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cc14a5-813a-44b8-9784-75d297818c43 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.328088 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cc14a5-813a-44b8-9784-75d297818c43 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.335997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce0407f-f2d3-49d2-a22c-7bdbaffbeb67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.337071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce0407f-f2d3-49d2-a22c-7bdbaffbeb67 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.337071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce0407f-f2d3-49d2-a22c-7bdbaffbeb67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.337494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce0407f-f2d3-49d2-a22c-7bdbaffbeb67 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.338034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce0407f-f2d3-49d2-a22c-7bdbaffbeb67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.339950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce0407f-f2d3-49d2-a22c-7bdbaffbeb67 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.353432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 788/1593] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:59 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:59.359232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 806/1594] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:07:59 2022] POST /identity/v3/users => generated 291 bytes in 307 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:59.360340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce0407f-f2d3-49d2-a22c-7bdbaffbeb67 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.360537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e08d91b-5a92-4986-a1f7-896680e2a06f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.360702 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e08d91b-5a92-4986-a1f7-896680e2a06f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.361144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e08d91b-5a92-4986-a1f7-896680e2a06f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.371413 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce0407f-f2d3-49d2-a22c-7bdbaffbeb67 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.377128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757a8f71-a226-4a00-b102-a98579b27930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.377869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757a8f71-a226-4a00-b102-a98579b27930 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.378700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757a8f71-a226-4a00-b102-a98579b27930 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.379011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757a8f71-a226-4a00-b102-a98579b27930 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.379757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757a8f71-a226-4a00-b102-a98579b27930 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.381435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757a8f71-a226-4a00-b102-a98579b27930 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.398465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 789/1595] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:59 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:59.398465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757a8f71-a226-4a00-b102-a98579b27930 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.409631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757a8f71-a226-4a00-b102-a98579b27930 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.413744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-2e08d91b-5a92-4986-a1f7-896680e2a06f None None] MFA Rules not processed for user `d8eae4b7123a445db9a3abace47ea29b`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:07:59.415016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39615074-c670-4cbf-b6b8-694b1fd7c296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.416249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2e08d91b-5a92-4986-a1f7-896680e2a06f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:07:59.416458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39615074-c670-4cbf-b6b8-694b1fd7c296 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.416458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39615074-c670-4cbf-b6b8-694b1fd7c296 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.416458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39615074-c670-4cbf-b6b8-694b1fd7c296 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.417018 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39615074-c670-4cbf-b6b8-694b1fd7c296 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.419456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39615074-c670-4cbf-b6b8-694b1fd7c296 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.452718 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 790/1596] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:07:59 2022] POST /identity/v3/users => generated 295 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:59.454491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39615074-c670-4cbf-b6b8-694b1fd7c296 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.471539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39615074-c670-4cbf-b6b8-694b1fd7c296 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.482125 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7597ee-de1b-4837-91a5-81602b8a56f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.483205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7597ee-de1b-4837-91a5-81602b8a56f1 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.483753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7597ee-de1b-4837-91a5-81602b8a56f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.484229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7597ee-de1b-4837-91a5-81602b8a56f1 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.485363 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7597ee-de1b-4837-91a5-81602b8a56f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.489347 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7597ee-de1b-4837-91a5-81602b8a56f1 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.531035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 791/1597] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:07:59 2022] POST /identity/v3/users => generated 283 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:07:59.533174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7597ee-de1b-4837-91a5-81602b8a56f1 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.549260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7597ee-de1b-4837-91a5-81602b8a56f1 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.556466 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 807/1598] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:07:59 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:07:59.557601 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c0fa27-826b-412e-95b3-f2cf9f0b5c9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.558288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e08d91b-5a92-4986-a1f7-896680e2a06f None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.558468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c0fa27-826b-412e-95b3-f2cf9f0b5c9e admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.558861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c0fa27-826b-412e-95b3-f2cf9f0b5c9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.559261 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c0fa27-826b-412e-95b3-f2cf9f0b5c9e admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.561292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c0fa27-826b-412e-95b3-f2cf9f0b5c9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.564567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c0fa27-826b-412e-95b3-f2cf9f0b5c9e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.583263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 792/1599] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:59 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:59.585211 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c0fa27-826b-412e-95b3-f2cf9f0b5c9e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.590527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e08d91b-5a92-4986-a1f7-896680e2a06f None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.603925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e75678e-5919-4fe8-9465-ddfee8fdf65b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.604112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c0fa27-826b-412e-95b3-f2cf9f0b5c9e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.604759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e75678e-5919-4fe8-9465-ddfee8fdf65b admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.605235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e75678e-5919-4fe8-9465-ddfee8fdf65b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.605730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e75678e-5919-4fe8-9465-ddfee8fdf65b admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.607059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e75678e-5919-4fe8-9465-ddfee8fdf65b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.611043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e75678e-5919-4fe8-9465-ddfee8fdf65b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.615398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b423e069-2c15-489d-b9f8-7aa74ef5544e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.616408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b423e069-2c15-489d-b9f8-7aa74ef5544e admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.616943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b423e069-2c15-489d-b9f8-7aa74ef5544e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.617379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b423e069-2c15-489d-b9f8-7aa74ef5544e admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.618735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b423e069-2c15-489d-b9f8-7aa74ef5544e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.621611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b423e069-2c15-489d-b9f8-7aa74ef5544e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.635107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 793/1600] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:59 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:59.635730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b423e069-2c15-489d-b9f8-7aa74ef5544e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.645725 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 808/1601] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:59 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:59.646513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b423e069-2c15-489d-b9f8-7aa74ef5544e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.647986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e75678e-5919-4fe8-9465-ddfee8fdf65b admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.653087 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be2204b-0c90-42f4-8118-43190c0a64cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.653872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be2204b-0c90-42f4-8118-43190c0a64cb admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.654071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be2204b-0c90-42f4-8118-43190c0a64cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.654329 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be2204b-0c90-42f4-8118-43190c0a64cb admin admin] PATH_INFO: `/v3/projects/b94a935febe14fc5b0bda4ca88f78ca8/users/20df8d74c87143dd81dbaae76b3c2b62/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.660728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:07:59.670237 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e75678e-5919-4fe8-9465-ddfee8fdf65b admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.675806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be2204b-0c90-42f4-8118-43190c0a64cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=b94a935febe14fc5b0bda4ca88f78ca8, user_id=20df8d74c87143dd81dbaae76b3c2b62, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.678581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be2204b-0c90-42f4-8118-43190c0a64cb admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.685111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79eaae3-6f52-4513-aa0e-522af18fd741 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.686224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79eaae3-6f52-4513-aa0e-522af18fd741 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.686732 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79eaae3-6f52-4513-aa0e-522af18fd741 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.687257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79eaae3-6f52-4513-aa0e-522af18fd741 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.688629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79eaae3-6f52-4513-aa0e-522af18fd741 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.692660 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79eaae3-6f52-4513-aa0e-522af18fd741 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.699706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 794/1602] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:07:59 2022] PUT /identity/v3/projects/b94a935febe14fc5b0bda4ca88f78ca8/users/20df8d74c87143dd81dbaae76b3c2b62/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:07:59.700826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be2204b-0c90-42f4-8118-43190c0a64cb admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.717698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 809/1603] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:59 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:59.719736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79eaae3-6f52-4513-aa0e-522af18fd741 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.723480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:INSERT=11 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:07:59.845294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be2204b-0c90-42f4-8118-43190c0a64cb admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.854940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c989ac1-d091-42cd-a56e-0ee764c65319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.856152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c989ac1-d091-42cd-a56e-0ee764c65319 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.856801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c989ac1-d091-42cd-a56e-0ee764c65319 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.857422 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c989ac1-d091-42cd-a56e-0ee764c65319 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.859041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c989ac1-d091-42cd-a56e-0ee764c65319 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.864446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c989ac1-d091-42cd-a56e-0ee764c65319 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.877275 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79eaae3-6f52-4513-aa0e-522af18fd741 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.902037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 795/1604] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:59 2022] GET /identity/v3/roles => generated 1420 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:59.905304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c989ac1-d091-42cd-a56e-0ee764c65319 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.939177 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c989ac1-d091-42cd-a56e-0ee764c65319 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:07:59.948829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a037c1ae-d2e6-478d-8ba6-befd5e491b0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.950002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a037c1ae-d2e6-478d-8ba6-befd5e491b0d admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.950329 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a037c1ae-d2e6-478d-8ba6-befd5e491b0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.950620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a037c1ae-d2e6-478d-8ba6-befd5e491b0d admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.951405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a037c1ae-d2e6-478d-8ba6-befd5e491b0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:07:59.953116 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a037c1ae-d2e6-478d-8ba6-befd5e491b0d admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:07:59.966413 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 796/1605] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:59 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:07:59.967251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a037c1ae-d2e6-478d-8ba6-befd5e491b0d admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:07:59.970686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399ba9ce-ef2e-48b4-8661-f6f97e6ac232 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:07:59.972906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399ba9ce-ef2e-48b4-8661-f6f97e6ac232 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:07:59.972906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399ba9ce-ef2e-48b4-8661-f6f97e6ac232 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:07:59.972906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399ba9ce-ef2e-48b4-8661-f6f97e6ac232 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:07:59.988598 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a037c1ae-d2e6-478d-8ba6-befd5e491b0d admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:00.000260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020a8cc2-2720-415d-91b6-19f9a5d12b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:00.001623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020a8cc2-2720-415d-91b6-19f9a5d12b51 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:00.003032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020a8cc2-2720-415d-91b6-19f9a5d12b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:00.003609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020a8cc2-2720-415d-91b6-19f9a5d12b51 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:00.004932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020a8cc2-2720-415d-91b6-19f9a5d12b51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:00.009244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020a8cc2-2720-415d-91b6-19f9a5d12b51 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:00.030563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 797/1606] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:07:59 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:00.030563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020a8cc2-2720-415d-91b6-19f9a5d12b51 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:00.044798 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020a8cc2-2720-415d-91b6-19f9a5d12b51 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:00.053547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280ad98b-9c55-402b-a0f0-d0da3eee8abd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:00.054436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280ad98b-9c55-402b-a0f0-d0da3eee8abd admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:00.056038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280ad98b-9c55-402b-a0f0-d0da3eee8abd admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:00.056038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280ad98b-9c55-402b-a0f0-d0da3eee8abd admin admin] PATH_INFO: `/v3/projects/be70baea277b4b34b677053f5810c337/users/3c537a32dd314556a29e7f3b47f0a874/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:00.083710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280ad98b-9c55-402b-a0f0-d0da3eee8abd admin admin] RBAC: Authorizing `identity:create_grant(project_id=be70baea277b4b34b677053f5810c337, user_id=3c537a32dd314556a29e7f3b47f0a874, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:00.087337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280ad98b-9c55-402b-a0f0-d0da3eee8abd admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:00.099583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399ba9ce-ef2e-48b4-8661-f6f97e6ac232 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:00.102535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399ba9ce-ef2e-48b4-8661-f6f97e6ac232 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:00.114795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 798/1607] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:00 2022] PUT /identity/v3/projects/be70baea277b4b34b677053f5810c337/users/3c537a32dd314556a29e7f3b47f0a874/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:00.117367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d21411d-6b55-4263-af20-eec4febe5d4b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:00.118080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d21411d-6b55-4263-af20-eec4febe5d4b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:00.118940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d21411d-6b55-4263-af20-eec4febe5d4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:00.137768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 810/1608] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:07:59 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:00.141079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399ba9ce-ef2e-48b4-8661-f6f97e6ac232 service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:00.167168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399ba9ce-ef2e-48b4-8661-f6f97e6ac232 service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:00.192833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6d21411d-6b55-4263-af20-eec4febe5d4b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:00.195024 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6d21411d-6b55-4263-af20-eec4febe5d4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:00.337664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf0cd33-6d70-4741-a28d-47286158c9fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:00.338508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf0cd33-6d70-4741-a28d-47286158c9fb admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:00.340044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf0cd33-6d70-4741-a28d-47286158c9fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:00.341426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf0cd33-6d70-4741-a28d-47286158c9fb admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:00.343528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf0cd33-6d70-4741-a28d-47286158c9fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:00.349922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf0cd33-6d70-4741-a28d-47286158c9fb admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:00.360206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:00.362649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 799/1609] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:00.364908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d21411d-6b55-4263-af20-eec4febe5d4b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:00.372158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 811/1610] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:00 2022] GET /identity/v3/roles => generated 1420 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:00.373314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf0cd33-6d70-4741-a28d-47286158c9fb admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:00.387512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf0cd33-6d70-4741-a28d-47286158c9fb admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:00.391485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d21411d-6b55-4263-af20-eec4febe5d4b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:00.393242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44119639-269d-4e00-a498-805948b6ac6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:00.395079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44119639-269d-4e00-a498-805948b6ac6e admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:00.395719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44119639-269d-4e00-a498-805948b6ac6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:00.396213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44119639-269d-4e00-a498-805948b6ac6e admin admin] PATH_INFO: `/v3/projects/8fa52072c8434e50af1a793687eba7fa/users/6e54d0633ea14df8a5487c6601486bb3/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:00.401917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d81af2a-41f2-46c0-9a82-3e18954539ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:00.402792 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d81af2a-41f2-46c0-9a82-3e18954539ce admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:00.403151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d81af2a-41f2-46c0-9a82-3e18954539ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:00.403539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d81af2a-41f2-46c0-9a82-3e18954539ce admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:00.404388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d81af2a-41f2-46c0-9a82-3e18954539ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:00.406476 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d81af2a-41f2-46c0-9a82-3e18954539ce admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:00.417902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44119639-269d-4e00-a498-805948b6ac6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fa52072c8434e50af1a793687eba7fa, user_id=6e54d0633ea14df8a5487c6601486bb3, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:00.421072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44119639-269d-4e00-a498-805948b6ac6e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:00.423445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 800/1611] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:00 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:00.425875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d81af2a-41f2-46c0-9a82-3e18954539ce admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:00.438986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d81af2a-41f2-46c0-9a82-3e18954539ce admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:00.445452 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 812/1612] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:00 2022] PUT /identity/v3/projects/8fa52072c8434e50af1a793687eba7fa/users/6e54d0633ea14df8a5487c6601486bb3/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:00.460252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f5ff9b-1e81-4d48-9ec6-bf913f5d5458 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:00.460252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f5ff9b-1e81-4d48-9ec6-bf913f5d5458 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:00.460252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f5ff9b-1e81-4d48-9ec6-bf913f5d5458 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:00.499574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a6f5ff9b-1e81-4d48-9ec6-bf913f5d5458 None None] MFA Rules not processed for user `3c537a32dd314556a29e7f3b47f0a874`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:00.501096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a6f5ff9b-1e81-4d48-9ec6-bf913f5d5458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:00.622012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 813/1613] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:00 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:00.623500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f5ff9b-1e81-4d48-9ec6-bf913f5d5458 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:00.658962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:00.705664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774d78c8-efc5-4af1-92b0-278776cfe450 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:00.709752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774d78c8-efc5-4af1-92b0-278776cfe450 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:00.710257 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774d78c8-efc5-4af1-92b0-278776cfe450 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:00.710776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774d78c8-efc5-4af1-92b0-278776cfe450 admin admin] PATH_INFO: `/v3/projects/470f049fa00841c98ce7422f3c5e423e/users/dfb41ea5723c416ea0969e214625b94b/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:00.796442 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774d78c8-efc5-4af1-92b0-278776cfe450 admin admin] RBAC: Authorizing `identity:create_grant(project_id=470f049fa00841c98ce7422f3c5e423e, user_id=dfb41ea5723c416ea0969e214625b94b, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:00.806546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774d78c8-efc5-4af1-92b0-278776cfe450 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:00.812547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f5ff9b-1e81-4d48-9ec6-bf913f5d5458 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:00.821513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee231aa-a696-43c7-a5a0-10b1a94e66ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:00.823141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee231aa-a696-43c7-a5a0-10b1a94e66ef admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:00.824272 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee231aa-a696-43c7-a5a0-10b1a94e66ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:00.827547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee231aa-a696-43c7-a5a0-10b1a94e66ef admin admin] PATH_INFO: `/v3/projects/80ea98b340d94181b2193f4a79905988/users/56732ebe8ee24f3f9ce8f87746e8b161/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:00.842406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 801/1614] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:00 2022] PUT /identity/v3/projects/470f049fa00841c98ce7422f3c5e423e/users/dfb41ea5723c416ea0969e214625b94b/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:00.842931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774d78c8-efc5-4af1-92b0-278776cfe450 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:00.865575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-774d78c8-efc5-4af1-92b0-278776cfe450 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:00.879679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee231aa-a696-43c7-a5a0-10b1a94e66ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=80ea98b340d94181b2193f4a79905988, user_id=56732ebe8ee24f3f9ce8f87746e8b161, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:00.889922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee231aa-a696-43c7-a5a0-10b1a94e66ef admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:00.935414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 814/1615] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:00 2022] PUT /identity/v3/projects/80ea98b340d94181b2193f4a79905988/users/56732ebe8ee24f3f9ce8f87746e8b161/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:00.941474 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee231aa-a696-43c7-a5a0-10b1a94e66ef admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:00.967419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774d78c8-efc5-4af1-92b0-278776cfe450 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:01.040049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d627cd-5fe2-4d86-8eb3-dfbfd4ee1408 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:01.041444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d627cd-5fe2-4d86-8eb3-dfbfd4ee1408 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:01.041859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d627cd-5fe2-4d86-8eb3-dfbfd4ee1408 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:01.042402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d627cd-5fe2-4d86-8eb3-dfbfd4ee1408 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:01.101028 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:01.128447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d627cd-5fe2-4d86-8eb3-dfbfd4ee1408 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:01.130327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d627cd-5fe2-4d86-8eb3-dfbfd4ee1408 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:01.135127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee231aa-a696-43c7-a5a0-10b1a94e66ef admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:01.142126 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca193a7-58ad-49a4-a30c-24f6e8b0749b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:01.144215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca193a7-58ad-49a4-a30c-24f6e8b0749b admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:01.144747 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca193a7-58ad-49a4-a30c-24f6e8b0749b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:01.145132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca193a7-58ad-49a4-a30c-24f6e8b0749b admin admin] PATH_INFO: `/v3/projects/4c82effc151742818916d6ff0431832b/users/2d6e967d04db459090cfdadc92e8703b/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:01.147064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 802/1616] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:01.148630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b905ad-ef6b-45d0-a6d2-85ffd505b9be None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:01.149027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b905ad-ef6b-45d0-a6d2-85ffd505b9be None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:01.149412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b905ad-ef6b-45d0-a6d2-85ffd505b9be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:01.184801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d4b905ad-ef6b-45d0-a6d2-85ffd505b9be None None] MFA Rules not processed for user `6e54d0633ea14df8a5487c6601486bb3`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:01.186060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca193a7-58ad-49a4-a30c-24f6e8b0749b admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c82effc151742818916d6ff0431832b, user_id=2d6e967d04db459090cfdadc92e8703b, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:01.187318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d4b905ad-ef6b-45d0-a6d2-85ffd505b9be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:01.199478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca193a7-58ad-49a4-a30c-24f6e8b0749b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:01.220306 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:01.255901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 815/1617] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:00 2022] PUT /identity/v3/projects/4c82effc151742818916d6ff0431832b/users/2d6e967d04db459090cfdadc92e8703b/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:01.259489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fe10b8-c362-4c53-a5a9-c5039628829f None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:01.260005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fe10b8-c362-4c53-a5a9-c5039628829f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:01.260500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fe10b8-c362-4c53-a5a9-c5039628829f None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:01.263182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 816/1618] 10.222.0.42 () {54 vars in 962 bytes} [Sun Jul 31 01:08:01 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:08:01.267861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bfe4c4-6c86-4783-bfbe-d2d9bb8aae97 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:01.270124 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bfe4c4-6c86-4783-bfbe-d2d9bb8aae97 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:01.272055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bfe4c4-6c86-4783-bfbe-d2d9bb8aae97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:01.301714 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 803/1619] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:01 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:01.301714 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c4e598-f8b3-4d0f-b31b-a2a7de74e38d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:01.301714 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c4e598-f8b3-4d0f-b31b-a2a7de74e38d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:01.301714 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c4e598-f8b3-4d0f-b31b-a2a7de74e38d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:01.330688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d4c4e598-f8b3-4d0f-b31b-a2a7de74e38d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:01.332923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d4c4e598-f8b3-4d0f-b31b-a2a7de74e38d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:01.400075 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f8bfe4c4-6c86-4783-bfbe-d2d9bb8aae97 None None] MFA Rules not processed for user `dfb41ea5723c416ea0969e214625b94b`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:01.401920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f8bfe4c4-6c86-4783-bfbe-d2d9bb8aae97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:01.431056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 804/1620] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:01.431056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62cff25-f3cf-486d-b482-da4d2ff0ba16 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:01.431056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62cff25-f3cf-486d-b482-da4d2ff0ba16 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:01.431056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62cff25-f3cf-486d-b482-da4d2ff0ba16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:01.511216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d62cff25-f3cf-486d-b482-da4d2ff0ba16 None None] MFA Rules not processed for user `2d6e967d04db459090cfdadc92e8703b`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:01.512533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d62cff25-f3cf-486d-b482-da4d2ff0ba16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:01.586051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 817/1621] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:01 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 322 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:01.587624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca31206-001d-4be3-93b8-57026ac67acc None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:01.587926 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca31206-001d-4be3-93b8-57026ac67acc None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:01.588180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca31206-001d-4be3-93b8-57026ac67acc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:01.741127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-5ca31206-001d-4be3-93b8-57026ac67acc None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:01.741127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5ca31206-001d-4be3-93b8-57026ac67acc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:01.804896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 818/1622] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Jul 31 01:08:01 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:01.807542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca31206-001d-4be3-93b8-57026ac67acc None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:01.838277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 805/1623] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:01 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 406 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:01.838277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62cff25-f3cf-486d-b482-da4d2ff0ba16 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:01.843905 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca31206-001d-4be3-93b8-57026ac67acc None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:01.860515 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31449cb0-5370-4c87-bb58-86ec435b2b94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:01.863056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31449cb0-5370-4c87-bb58-86ec435b2b94 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:01.863335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31449cb0-5370-4c87-bb58-86ec435b2b94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:01.867464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31449cb0-5370-4c87-bb58-86ec435b2b94 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:01.869146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31449cb0-5370-4c87-bb58-86ec435b2b94 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:01.875950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31449cb0-5370-4c87-bb58-86ec435b2b94 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:02.052478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 819/1624] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:01 2022] POST /identity/v3/projects => generated 405 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:02.057625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31449cb0-5370-4c87-bb58-86ec435b2b94 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:02.085447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62cff25-f3cf-486d-b482-da4d2ff0ba16 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:02.096554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:02.109289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:02.226471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31449cb0-5370-4c87-bb58-86ec435b2b94 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:02.238790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf419796-aed8-4aa3-93dd-e1ab7f72fdf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:02.239506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf419796-aed8-4aa3-93dd-e1ab7f72fdf2 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:02.239753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf419796-aed8-4aa3-93dd-e1ab7f72fdf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:02.240026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf419796-aed8-4aa3-93dd-e1ab7f72fdf2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:02.240730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf419796-aed8-4aa3-93dd-e1ab7f72fdf2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:02.242748 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf419796-aed8-4aa3-93dd-e1ab7f72fdf2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:02.281210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66255d7b-9090-486d-bf96-e99d4f828269 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:02.285996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66255d7b-9090-486d-bf96-e99d4f828269 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:02.291128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66255d7b-9090-486d-bf96-e99d4f828269 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:02.299282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66255d7b-9090-486d-bf96-e99d4f828269 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:02.314830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 820/1625] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:02 2022] POST /identity/v3/projects => generated 381 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:02.316335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf419796-aed8-4aa3-93dd-e1ab7f72fdf2 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:02.351201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66255d7b-9090-486d-bf96-e99d4f828269 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:02.355468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66255d7b-9090-486d-bf96-e99d4f828269 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:02.392278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 806/1626] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 557 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:02.393501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66255d7b-9090-486d-bf96-e99d4f828269 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:02.440832 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf419796-aed8-4aa3-93dd-e1ab7f72fdf2 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:02.460961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6336a47b-bb1e-4ad8-921e-27524fc6d037 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:02.461712 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6336a47b-bb1e-4ad8-921e-27524fc6d037 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:02.461994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6336a47b-bb1e-4ad8-921e-27524fc6d037 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:02.462270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6336a47b-bb1e-4ad8-921e-27524fc6d037 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:02.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66255d7b-9090-486d-bf96-e99d4f828269 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:02.635346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6336a47b-bb1e-4ad8-921e-27524fc6d037 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:02.637438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6336a47b-bb1e-4ad8-921e-27524fc6d037 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:02.643793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5a3022-3836-419e-898d-46ec7659a5f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:02.644780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a3022-3836-419e-898d-46ec7659a5f3 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:02.645071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a3022-3836-419e-898d-46ec7659a5f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:02.648432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a3022-3836-419e-898d-46ec7659a5f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:02.660018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 821/1627] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:02 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:02.661219 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6336a47b-bb1e-4ad8-921e-27524fc6d037 service nova] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:02.717561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:02.779956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6336a47b-bb1e-4ad8-921e-27524fc6d037 service nova] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:02.802582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe04bf-3c35-4ef6-b1d9-500e56c8ca90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:02.803568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe04bf-3c35-4ef6-b1d9-500e56c8ca90 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:02.805751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe04bf-3c35-4ef6-b1d9-500e56c8ca90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:02.808836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fe04bf-3c35-4ef6-b1d9-500e56c8ca90 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:02.811460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fe04bf-3c35-4ef6-b1d9-500e56c8ca90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:02.819919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fe04bf-3c35-4ef6-b1d9-500e56c8ca90 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:02.837297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5a3022-3836-419e-898d-46ec7659a5f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:02.841986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5a3022-3836-419e-898d-46ec7659a5f3 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:02.882756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 807/1628] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:02 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:02.902405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5a3022-3836-419e-898d-46ec7659a5f3 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:02.911631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 822/1629] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:02 2022] POST /identity/v3/users => generated 309 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:02.918011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe04bf-3c35-4ef6-b1d9-500e56c8ca90 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:02.956526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5a3022-3836-419e-898d-46ec7659a5f3 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:02.962192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fe04bf-3c35-4ef6-b1d9-500e56c8ca90 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:02.984277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0258bb-36e9-423f-8d4a-6355344265b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:02.994871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0258bb-36e9-423f-8d4a-6355344265b4 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:02.995206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0258bb-36e9-423f-8d4a-6355344265b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:02.995497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0258bb-36e9-423f-8d4a-6355344265b4 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:03.000218 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0258bb-36e9-423f-8d4a-6355344265b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:03.001073 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b20cc0-80b2-49b5-8183-f02bf24fb70e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:03.001732 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b20cc0-80b2-49b5-8183-f02bf24fb70e admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:03.002010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b20cc0-80b2-49b5-8183-f02bf24fb70e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:03.002282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b20cc0-80b2-49b5-8183-f02bf24fb70e admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:03.004962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0258bb-36e9-423f-8d4a-6355344265b4 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:03.010622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b20cc0-80b2-49b5-8183-f02bf24fb70e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:03.014097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b20cc0-80b2-49b5-8183-f02bf24fb70e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:03.049598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 823/1630] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:02 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:03.058212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b20cc0-80b2-49b5-8183-f02bf24fb70e admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:03.078726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 808/1631] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:02 2022] POST /identity/v3/users => generated 297 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:03.083740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0258bb-36e9-423f-8d4a-6355344265b4 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:03.102750 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b20cc0-80b2-49b5-8183-f02bf24fb70e admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:03.116697 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca4e41c-481b-4627-a2cf-d6b7dbbf6b0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:03.117400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca4e41c-481b-4627-a2cf-d6b7dbbf6b0d admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:03.117681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca4e41c-481b-4627-a2cf-d6b7dbbf6b0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:03.117959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca4e41c-481b-4627-a2cf-d6b7dbbf6b0d admin admin] PATH_INFO: `/v3/projects/a6e86637a7694247b50ef535d2a4984d/users/8b86fc747fdb4a8285e54e6bcac9a0e0/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:03.130217 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0258bb-36e9-423f-8d4a-6355344265b4 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:03.149917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1c4cc3-d6fa-4235-8b2d-afe86a38e1c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:03.151314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1c4cc3-d6fa-4235-8b2d-afe86a38e1c2 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:03.151853 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1c4cc3-d6fa-4235-8b2d-afe86a38e1c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:03.152484 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1c4cc3-d6fa-4235-8b2d-afe86a38e1c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:03.156799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca4e41c-481b-4627-a2cf-d6b7dbbf6b0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6e86637a7694247b50ef535d2a4984d, user_id=8b86fc747fdb4a8285e54e6bcac9a0e0, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:03.157733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1c4cc3-d6fa-4235-8b2d-afe86a38e1c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:03.160081 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca4e41c-481b-4627-a2cf-d6b7dbbf6b0d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:03.160998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1c4cc3-d6fa-4235-8b2d-afe86a38e1c2 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:03.163205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:03.180565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 809/1632] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:03 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:03.187887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1c4cc3-d6fa-4235-8b2d-afe86a38e1c2 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:03.188358 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 824/1633] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:03 2022] PUT /identity/v3/projects/a6e86637a7694247b50ef535d2a4984d/users/8b86fc747fdb4a8285e54e6bcac9a0e0/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:03.193192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca4e41c-481b-4627-a2cf-d6b7dbbf6b0d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:03.231424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1c4cc3-d6fa-4235-8b2d-afe86a38e1c2 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:03.345880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca4e41c-481b-4627-a2cf-d6b7dbbf6b0d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:03.370026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53298f5b-f9e8-45fe-a127-8c50718a35ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:03.370687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53298f5b-f9e8-45fe-a127-8c50718a35ff admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:03.370975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53298f5b-f9e8-45fe-a127-8c50718a35ff admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:03.371250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53298f5b-f9e8-45fe-a127-8c50718a35ff admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:03.372007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53298f5b-f9e8-45fe-a127-8c50718a35ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:03.373762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53298f5b-f9e8-45fe-a127-8c50718a35ff admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:03.397191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a5bf12-7721-4207-8495-35e74008f8e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:03.400696 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 825/1634] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:03 2022] GET /identity/v3/roles => generated 1420 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:03.401125 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a5bf12-7721-4207-8495-35e74008f8e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:03.401559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a5bf12-7721-4207-8495-35e74008f8e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:03.402012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a5bf12-7721-4207-8495-35e74008f8e0 admin admin] PATH_INFO: `/v3/projects/7ad4673ebcbc464b802a95670bb03f45/users/e9cff3b66be64f4cb3ad7e88a6e88267/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:03.404879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53298f5b-f9e8-45fe-a127-8c50718a35ff admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:03.428954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53298f5b-f9e8-45fe-a127-8c50718a35ff admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:03.437334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972f0b67-e5fa-4da4-afb8-cb417bce2d79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:03.438506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f0b67-e5fa-4da4-afb8-cb417bce2d79 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:03.438969 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f0b67-e5fa-4da4-afb8-cb417bce2d79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:03.439476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f0b67-e5fa-4da4-afb8-cb417bce2d79 admin admin] PATH_INFO: `/v3/projects/a6e86637a7694247b50ef535d2a4984d/users/8b86fc747fdb4a8285e54e6bcac9a0e0/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:03.449528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972f0b67-e5fa-4da4-afb8-cb417bce2d79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6e86637a7694247b50ef535d2a4984d, user_id=8b86fc747fdb4a8285e54e6bcac9a0e0, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:03.455789 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972f0b67-e5fa-4da4-afb8-cb417bce2d79 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:03.467245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a5bf12-7721-4207-8495-35e74008f8e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ad4673ebcbc464b802a95670bb03f45, user_id=e9cff3b66be64f4cb3ad7e88a6e88267, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:03.476147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a5bf12-7721-4207-8495-35e74008f8e0 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:03.482638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 826/1635] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:03 2022] PUT /identity/v3/projects/a6e86637a7694247b50ef535d2a4984d/users/8b86fc747fdb4a8285e54e6bcac9a0e0/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:03.509375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd8ce0e-6096-4c9d-ab52-deb1f3bcf15d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:03.510220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 810/1636] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:03 2022] PUT /identity/v3/projects/7ad4673ebcbc464b802a95670bb03f45/users/e9cff3b66be64f4cb3ad7e88a6e88267/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:03.511334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd8ce0e-6096-4c9d-ab52-deb1f3bcf15d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:03.511626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd8ce0e-6096-4c9d-ab52-deb1f3bcf15d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:03.514064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a5bf12-7721-4207-8495-35e74008f8e0 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:03.686195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-2dd8ce0e-6096-4c9d-ab52-deb1f3bcf15d None None] MFA Rules not processed for user `8b86fc747fdb4a8285e54e6bcac9a0e0`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:03.690035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2dd8ce0e-6096-4c9d-ab52-deb1f3bcf15d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:03.806032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a5bf12-7721-4207-8495-35e74008f8e0 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:03.837710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18214f64-ac8d-44ea-9565-1deab3dd1acc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:03.838415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18214f64-ac8d-44ea-9565-1deab3dd1acc admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:03.838699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18214f64-ac8d-44ea-9565-1deab3dd1acc admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:03.838974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18214f64-ac8d-44ea-9565-1deab3dd1acc admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:03.839818 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18214f64-ac8d-44ea-9565-1deab3dd1acc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:03.841522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18214f64-ac8d-44ea-9565-1deab3dd1acc admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:03.878934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 811/1637] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:03 2022] GET /identity/v3/roles => generated 1420 bytes in 365 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:03.892413 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18214f64-ac8d-44ea-9565-1deab3dd1acc admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:03.923426 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18214f64-ac8d-44ea-9565-1deab3dd1acc admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:03.931706 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:03.940978 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3b315b-d589-4095-a742-b9ec0c35158b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:03.941787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3b315b-d589-4095-a742-b9ec0c35158b admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:03.942068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3b315b-d589-4095-a742-b9ec0c35158b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:03.942345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3b315b-d589-4095-a742-b9ec0c35158b admin admin] PATH_INFO: `/v3/projects/7ad4673ebcbc464b802a95670bb03f45/users/e9cff3b66be64f4cb3ad7e88a6e88267/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:03.953621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 827/1638] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:03 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 454 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:03.953909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3b315b-d589-4095-a742-b9ec0c35158b admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ad4673ebcbc464b802a95670bb03f45, user_id=e9cff3b66be64f4cb3ad7e88a6e88267, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:03.960082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3b315b-d589-4095-a742-b9ec0c35158b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:03.996640 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 812/1639] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:03 2022] PUT /identity/v3/projects/7ad4673ebcbc464b802a95670bb03f45/users/e9cff3b66be64f4cb3ad7e88a6e88267/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:04.013897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ce5311-7d89-4101-9a79-274f77a08c55 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:04.013897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ce5311-7d89-4101-9a79-274f77a08c55 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:04.013897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ce5311-7d89-4101-9a79-274f77a08c55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:04.073224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-78ce5311-7d89-4101-9a79-274f77a08c55 None None] MFA Rules not processed for user `e9cff3b66be64f4cb3ad7e88a6e88267`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:04.075196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-78ce5311-7d89-4101-9a79-274f77a08c55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:04.088666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3b315b-d589-4095-a742-b9ec0c35158b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:04.211608 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3b315b-d589-4095-a742-b9ec0c35158b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:04.222577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a79a5e3-5180-4c5a-964e-78efd98af4a9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:04.223120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a79a5e3-5180-4c5a-964e-78efd98af4a9 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:04.223503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a79a5e3-5180-4c5a-964e-78efd98af4a9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:04.224152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a79a5e3-5180-4c5a-964e-78efd98af4a9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:04.271978 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:04.330366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 828/1640] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:04 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:04.379402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bf9350-ec9f-41ad-9adb-79ba1c70ba52 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:04.380079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bf9350-ec9f-41ad-9adb-79ba1c70ba52 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:04.380079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bf9350-ec9f-41ad-9adb-79ba1c70ba52 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:04.381985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 829/1641] 10.222.0.42 () {54 vars in 962 bytes} [Sun Jul 31 01:08:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 01:08:04.414635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a79a5e3-5180-4c5a-964e-78efd98af4a9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:04.417109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dde98ae-8b09-49c5-8201-ef240448fde8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:04.417328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dde98ae-8b09-49c5-8201-ef240448fde8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:04.417482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dde98ae-8b09-49c5-8201-ef240448fde8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:04.420635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a79a5e3-5180-4c5a-964e-78efd98af4a9 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:04.467497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 813/1642] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:04 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:08:04.476592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-7dde98ae-8b09-49c5-8201-ef240448fde8 None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:04.479732 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7dde98ae-8b09-49c5-8201-ef240448fde8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:04.684810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a79a5e3-5180-4c5a-964e-78efd98af4a9 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:04.696148 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a79a5e3-5180-4c5a-964e-78efd98af4a9 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:04.705214 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f748a1c3-b74b-4296-b65c-4855cccf4a90 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:04.707058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f748a1c3-b74b-4296-b65c-4855cccf4a90 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:04.707058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f748a1c3-b74b-4296-b65c-4855cccf4a90 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:04.707545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f748a1c3-b74b-4296-b65c-4855cccf4a90 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:04.719088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 830/1643] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Jul 31 01:08:04 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 303 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:04.727818 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f748a1c3-b74b-4296-b65c-4855cccf4a90 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:04.730248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f748a1c3-b74b-4296-b65c-4855cccf4a90 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:04.735741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b41f78-585e-4425-8bc9-b6ad31d3e9ae None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:04.736148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b41f78-585e-4425-8bc9-b6ad31d3e9ae None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:04.736148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b41f78-585e-4425-8bc9-b6ad31d3e9ae None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:04.738343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 831/1644] 10.222.0.42 () {54 vars in 912 bytes} [Sun Jul 31 01:08:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 01:08:04.745873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 814/1645] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:04 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:08:04.756815 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63571cbc-a845-4a38-9311-f9d91d736dc3 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:04.757079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63571cbc-a845-4a38-9311-f9d91d736dc3 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:04.757357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63571cbc-a845-4a38-9311-f9d91d736dc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:04.805345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-63571cbc-a845-4a38-9311-f9d91d736dc3 None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:04.817525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-63571cbc-a845-4a38-9311-f9d91d736dc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:04.853973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 832/1646] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:08:04 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:04.983313 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f748a1c3-b74b-4296-b65c-4855cccf4a90 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:04.997266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f748a1c3-b74b-4296-b65c-4855cccf4a90 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:05.007161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166d514-618a-496c-821e-eb012778dcee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:05.009206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166d514-618a-496c-821e-eb012778dcee service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:05.009358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166d514-618a-496c-821e-eb012778dcee service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:05.009630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166d514-618a-496c-821e-eb012778dcee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:05.035147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166d514-618a-496c-821e-eb012778dcee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:05.037284 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166d514-618a-496c-821e-eb012778dcee service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:05.052908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 815/1647] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:04 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:08:05.107693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63571cbc-a845-4a38-9311-f9d91d736dc3 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:05.119600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63571cbc-a845-4a38-9311-f9d91d736dc3 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:05.130705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24a3ac6-9f23-4693-af33-7fc46d365283 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:05.131392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24a3ac6-9f23-4693-af33-7fc46d365283 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:05.131835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24a3ac6-9f23-4693-af33-7fc46d365283 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:05.132005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24a3ac6-9f23-4693-af33-7fc46d365283 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:05.166399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166d514-618a-496c-821e-eb012778dcee service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:05.336606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166d514-618a-496c-821e-eb012778dcee service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:05.342585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c5a62f-2caf-4d9a-8b9a-2c0b96d9aa7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:05.343766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c5a62f-2caf-4d9a-8b9a-2c0b96d9aa7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:05.343766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c5a62f-2caf-4d9a-8b9a-2c0b96d9aa7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:05.343998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c5a62f-2caf-4d9a-8b9a-2c0b96d9aa7e admin admin] PATH_INFO: `/v3/users/cf8d97711043403b95da4c0d1fc6942f` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:05.345440 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c5a62f-2caf-4d9a-8b9a-2c0b96d9aa7e admin admin] RBAC: Authorizing `identity:delete_user(user_id=cf8d97711043403b95da4c0d1fc6942f)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:05.346846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: WARNING py.warnings [None req-f3c5a62f-2caf-4d9a-8b9a-2c0b96d9aa7e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:08:05.346846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: warnings.warn(msg) Jul 31 01:08:05.346846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: Jul 31 01:08:05.348935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c5a62f-2caf-4d9a-8b9a-2c0b96d9aa7e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:05.362716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24a3ac6-9f23-4693-af33-7fc46d365283 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:05.370922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24a3ac6-9f23-4693-af33-7fc46d365283 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:05.400483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 833/1648] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:05 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:05.401515 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24a3ac6-9f23-4693-af33-7fc46d365283 service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:05.446728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24a3ac6-9f23-4693-af33-7fc46d365283 service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:05.451995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4ab638-7b25-4040-9c2b-f203079d6024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:05.452722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4ab638-7b25-4040-9c2b-f203079d6024 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:05.453003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4ab638-7b25-4040-9c2b-f203079d6024 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:05.453281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4ab638-7b25-4040-9c2b-f203079d6024 admin admin] PATH_INFO: `/v3/users/6e54d0633ea14df8a5487c6601486bb3` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:05.454600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4ab638-7b25-4040-9c2b-f203079d6024 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e54d0633ea14df8a5487c6601486bb3)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:05.457578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4ab638-7b25-4040-9c2b-f203079d6024 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:05.467096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:05.496603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-f3c5a62f-2caf-4d9a-8b9a-2c0b96d9aa7e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cf8d97711043403b95da4c0d1fc6942f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:05.516580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-f3c5a62f-2caf-4d9a-8b9a-2c0b96d9aa7e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cf8d97711043403b95da4c0d1fc6942f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:05.517348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-f3c5a62f-2caf-4d9a-8b9a-2c0b96d9aa7e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cf8d97711043403b95da4c0d1fc6942f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:05.573624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-f3c5a62f-2caf-4d9a-8b9a-2c0b96d9aa7e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cf8d97711043403b95da4c0d1fc6942f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:05.578753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-f3c5a62f-2caf-4d9a-8b9a-2c0b96d9aa7e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cf8d97711043403b95da4c0d1fc6942f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:05.579562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8e4ab638-7b25-4040-9c2b-f203079d6024 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e54d0633ea14df8a5487c6601486bb3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:05.580285 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8e4ab638-7b25-4040-9c2b-f203079d6024 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e54d0633ea14df8a5487c6601486bb3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:05.594974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8e4ab638-7b25-4040-9c2b-f203079d6024 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e54d0633ea14df8a5487c6601486bb3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:05.620855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8e4ab638-7b25-4040-9c2b-f203079d6024 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e54d0633ea14df8a5487c6601486bb3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:05.630281 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 816/1649] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:05 2022] DELETE /identity/v3/users/cf8d97711043403b95da4c0d1fc6942f => generated 0 bytes in 461 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:05.633296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c96cd64-df6f-4ce7-b972-28dad565555a None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:05.635328 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c96cd64-df6f-4ce7-b972-28dad565555a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:05.635779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c96cd64-df6f-4ce7-b972-28dad565555a None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:05.638390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8e4ab638-7b25-4040-9c2b-f203079d6024 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e54d0633ea14df8a5487c6601486bb3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:05.640084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 817/1650] 10.222.0.42 () {54 vars in 912 bytes} [Sun Jul 31 01:08:05 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:08:05.645948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cdc9db-5bc6-4841-86ca-8be3e42c046e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:05.645948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cdc9db-5bc6-4841-86ca-8be3e42c046e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:05.646238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cdc9db-5bc6-4841-86ca-8be3e42c046e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:05.657472 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 834/1651] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:05 2022] DELETE /identity/v3/users/6e54d0633ea14df8a5487c6601486bb3 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:05.658602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e364c4e-32db-428f-9613-b7d063a542f8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:05.659249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e364c4e-32db-428f-9613-b7d063a542f8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:05.660555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e364c4e-32db-428f-9613-b7d063a542f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:05.667690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:05.698301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-88cdc9db-5bc6-4841-86ca-8be3e42c046e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:05.698577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-88cdc9db-5bc6-4841-86ca-8be3e42c046e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:05.704819 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-8e364c4e-32db-428f-9613-b7d063a542f8 None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:05.706214 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8e364c4e-32db-428f-9613-b7d063a542f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:05.876705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 818/1652] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:05.883177 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c95ad8-3c13-41cb-85af-93c6ae9f7125 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:05.883739 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c95ad8-3c13-41cb-85af-93c6ae9f7125 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:05.884122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c95ad8-3c13-41cb-85af-93c6ae9f7125 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:05.917561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-64c95ad8-3c13-41cb-85af-93c6ae9f7125 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:05.919141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-64c95ad8-3c13-41cb-85af-93c6ae9f7125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:05.936675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 819/1653] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:05.956171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 835/1654] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:08:05 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:05.968445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c95ad8-3c13-41cb-85af-93c6ae9f7125 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:05.970531 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-64c95ad8-3c13-41cb-85af-93c6ae9f7125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:05.981311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e364c4e-32db-428f-9613-b7d063a542f8 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:05.983030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8e364c4e-32db-428f-9613-b7d063a542f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:06.148307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e364c4e-32db-428f-9613-b7d063a542f8 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:06.161472 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b056b8c-ab10-43b5-8ebd-87be31f551c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:06.162133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b056b8c-ab10-43b5-8ebd-87be31f551c5 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:06.162419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b056b8c-ab10-43b5-8ebd-87be31f551c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:06.162699 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b056b8c-ab10-43b5-8ebd-87be31f551c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:06.185004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b056b8c-ab10-43b5-8ebd-87be31f551c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:06.192316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b056b8c-ab10-43b5-8ebd-87be31f551c5 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:06.201077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c95ad8-3c13-41cb-85af-93c6ae9f7125 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:06.218663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3828869f-dae8-465f-bdde-f448cc0e8924 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:06.219316 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3828869f-dae8-465f-bdde-f448cc0e8924 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:06.219632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3828869f-dae8-465f-bdde-f448cc0e8924 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:06.219908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3828869f-dae8-465f-bdde-f448cc0e8924 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:06.220621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 836/1655] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:06.222282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b056b8c-ab10-43b5-8ebd-87be31f551c5 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:06.228360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9b056b8c-ab10-43b5-8ebd-87be31f551c5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:06.246063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3828869f-dae8-465f-bdde-f448cc0e8924 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:06.247605 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3828869f-dae8-465f-bdde-f448cc0e8924 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:06.259122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b056b8c-ab10-43b5-8ebd-87be31f551c5 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:06.267570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52255167-48af-44bf-a0fc-4cda6de5b9c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:06.269422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52255167-48af-44bf-a0fc-4cda6de5b9c0 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:06.270372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52255167-48af-44bf-a0fc-4cda6de5b9c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:06.271523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52255167-48af-44bf-a0fc-4cda6de5b9c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:06.286042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 820/1656] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:05 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:06.303600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52255167-48af-44bf-a0fc-4cda6de5b9c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:06.307292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52255167-48af-44bf-a0fc-4cda6de5b9c0 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:06.322744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:06.341641 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 837/1657] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:06.443335 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3828869f-dae8-465f-bdde-f448cc0e8924 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:06.444871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-3828869f-dae8-465f-bdde-f448cc0e8924 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:06.471227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3828869f-dae8-465f-bdde-f448cc0e8924 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:06.487781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:06.488812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:06.488934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:06.489146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] PATH_INFO: `/v3/users/3c537a32dd314556a29e7f3b47f0a874` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:06.490482 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c537a32dd314556a29e7f3b47f0a874)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:06.496559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:06.622817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c537a32dd314556a29e7f3b47f0a874'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:06.633384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c537a32dd314556a29e7f3b47f0a874'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:06.634739 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c537a32dd314556a29e7f3b47f0a874'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:06.693836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c537a32dd314556a29e7f3b47f0a874'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:06.695215 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c537a32dd314556a29e7f3b47f0a874'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:06.722802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 821/1658] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:06 2022] DELETE /identity/v3/users/3c537a32dd314556a29e7f3b47f0a874 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:06.727960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4800f3ae-9359-469c-b7ca-4034382c09a5 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:06.729388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4800f3ae-9359-469c-b7ca-4034382c09a5 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:06.730332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4800f3ae-9359-469c-b7ca-4034382c09a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:06.817988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-4800f3ae-9359-469c-b7ca-4034382c09a5 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:06.818685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4800f3ae-9359-469c-b7ca-4034382c09a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:07.012097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 838/1659] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:07.040421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:07.041919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:07.080244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:07.240997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff3bd9d-20a5-4895-85a2-00e024b4efb2 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:07.254521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0035c3-a6a1-4db2-9ff3-c2cb27a07cb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:07.255176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0035c3-a6a1-4db2-9ff3-c2cb27a07cb7 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:07.255499 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0035c3-a6a1-4db2-9ff3-c2cb27a07cb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:07.256409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0035c3-a6a1-4db2-9ff3-c2cb27a07cb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:07.281516 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0035c3-a6a1-4db2-9ff3-c2cb27a07cb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:07.283090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0035c3-a6a1-4db2-9ff3-c2cb27a07cb7 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:07.322459 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 822/1660] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:07.912471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4800f3ae-9359-469c-b7ca-4034382c09a5 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:07.914717 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4800f3ae-9359-469c-b7ca-4034382c09a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:07.985900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4800f3ae-9359-469c-b7ca-4034382c09a5 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:08.000535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:08.006379 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:08.006901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:08.007264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] PATH_INFO: `/v3/projects/8fa52072c8434e50af1a793687eba7fa` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:08.011580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=8fa52072c8434e50af1a793687eba7fa)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:08.013023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: WARNING py.warnings [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 31 01:08:08.013023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: warnings.warn(msg) Jul 31 01:08:08.013023 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: Jul 31 01:08:08.017328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:08.183804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8fa52072c8434e50af1a793687eba7fa', 'request_id': 'req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:08.224610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8fa52072c8434e50af1a793687eba7fa', 'request_id': 'req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:08.250687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8fa52072c8434e50af1a793687eba7fa', 'request_id': 'req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:08.303089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] The token cache is being invalidate because project 8fa52072c8434e50af1a793687eba7fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:08.303755 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:08.306524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:08.311910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 839/1661] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:07 2022] DELETE /identity/v3/projects/8fa52072c8434e50af1a793687eba7fa => generated 0 bytes in 400 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:08.317509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0035c3-a6a1-4db2-9ff3-c2cb27a07cb7 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:08.363569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0035c3-a6a1-4db2-9ff3-c2cb27a07cb7 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:08.505476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:08.507414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:08.744761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:08.746007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:08.746483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:08.746960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] PATH_INFO: `/v3/users/8b86fc747fdb4a8285e54e6bcac9a0e0` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:08.749575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b86fc747fdb4a8285e54e6bcac9a0e0)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:08.754902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:09.021008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b86fc747fdb4a8285e54e6bcac9a0e0'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:09.023648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf09356d-ac41-4dc4-8b58-145d2c23e6cf admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:09.052711 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:09.057142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:09.059848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:09.060500 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b86fc747fdb4a8285e54e6bcac9a0e0'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:09.064918 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b86fc747fdb4a8285e54e6bcac9a0e0'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:09.066017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] PATH_INFO: `/v3/projects/e29bf692ac1845bbac73e28693875517` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:09.074230 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e29bf692ac1845bbac73e28693875517)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:09.082927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:09.116610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b86fc747fdb4a8285e54e6bcac9a0e0'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:09.118068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b86fc747fdb4a8285e54e6bcac9a0e0'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:09.161108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 823/1662] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:08 2022] DELETE /identity/v3/users/8b86fc747fdb4a8285e54e6bcac9a0e0 => generated 0 bytes in 845 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:09.241386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:09.282081 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e29bf692ac1845bbac73e28693875517', 'request_id': 'req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:09.291024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e29bf692ac1845bbac73e28693875517', 'request_id': 'req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:09.306061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e29bf692ac1845bbac73e28693875517', 'request_id': 'req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:09.334282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] The token cache is being invalidate because project e29bf692ac1845bbac73e28693875517 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:09.334730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:09.338342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:09.341771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 840/1663] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:08 2022] DELETE /identity/v3/projects/e29bf692ac1845bbac73e28693875517 => generated 0 bytes in 837 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:09.347658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:09.350097 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:09.524132 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:09.635082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42231f46-d835-4e63-8655-0d4c82fbd19d admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:09.881003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b40089a-abcc-4fc9-9002-f93a51ac2a81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:09.882189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b40089a-abcc-4fc9-9002-f93a51ac2a81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:09.882693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b40089a-abcc-4fc9-9002-f93a51ac2a81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:09.883210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b40089a-abcc-4fc9-9002-f93a51ac2a81 admin admin] PATH_INFO: `/v3/users/d8eae4b7123a445db9a3abace47ea29b` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:09.886252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b40089a-abcc-4fc9-9002-f93a51ac2a81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8eae4b7123a445db9a3abace47ea29b)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:09.890191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b40089a-abcc-4fc9-9002-f93a51ac2a81 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:10.060694 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0b40089a-abcc-4fc9-9002-f93a51ac2a81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8eae4b7123a445db9a3abace47ea29b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:10.093221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0b40089a-abcc-4fc9-9002-f93a51ac2a81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8eae4b7123a445db9a3abace47ea29b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:10.096957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0b40089a-abcc-4fc9-9002-f93a51ac2a81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8eae4b7123a445db9a3abace47ea29b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:10.147759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0b40089a-abcc-4fc9-9002-f93a51ac2a81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8eae4b7123a445db9a3abace47ea29b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:10.152775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0b40089a-abcc-4fc9-9002-f93a51ac2a81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8eae4b7123a445db9a3abace47ea29b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:10.198552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 824/1664] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:09 2022] DELETE /identity/v3/users/d8eae4b7123a445db9a3abace47ea29b => generated 0 bytes in 852 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:11.229093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:11.231311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:11.262241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b62ee0-54d7-4b21-93fb-27a2b61d0e7f admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:11.470707 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0800f8af-0f3c-46f2-824d-f4f51671993a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:11.472142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0800f8af-0f3c-46f2-824d-f4f51671993a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:11.472425 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0800f8af-0f3c-46f2-824d-f4f51671993a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:11.472854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0800f8af-0f3c-46f2-824d-f4f51671993a admin admin] PATH_INFO: `/v3/projects/be70baea277b4b34b677053f5810c337` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:11.474518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0800f8af-0f3c-46f2-824d-f4f51671993a admin admin] RBAC: Authorizing `identity:delete_project(project_id=be70baea277b4b34b677053f5810c337)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:11.479472 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0800f8af-0f3c-46f2-824d-f4f51671993a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:11.618505 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0800f8af-0f3c-46f2-824d-f4f51671993a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be70baea277b4b34b677053f5810c337', 'request_id': 'req-0800f8af-0f3c-46f2-824d-f4f51671993a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:11.643961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0800f8af-0f3c-46f2-824d-f4f51671993a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be70baea277b4b34b677053f5810c337', 'request_id': 'req-0800f8af-0f3c-46f2-824d-f4f51671993a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:11.675398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0800f8af-0f3c-46f2-824d-f4f51671993a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be70baea277b4b34b677053f5810c337', 'request_id': 'req-0800f8af-0f3c-46f2-824d-f4f51671993a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:11.714675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0800f8af-0f3c-46f2-824d-f4f51671993a admin admin] The token cache is being invalidate because project be70baea277b4b34b677053f5810c337 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:11.723967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0800f8af-0f3c-46f2-824d-f4f51671993a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:11.724701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0800f8af-0f3c-46f2-824d-f4f51671993a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:11.726839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 841/1665] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:11 2022] DELETE /identity/v3/projects/be70baea277b4b34b677053f5810c337 => generated 0 bytes in 498 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:11.733688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19fb0fc-fbf2-43c3-afe9-d60de4d300cd None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:11.734399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19fb0fc-fbf2-43c3-afe9-d60de4d300cd None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:11.734399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19fb0fc-fbf2-43c3-afe9-d60de4d300cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:11.766053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-f19fb0fc-fbf2-43c3-afe9-d60de4d300cd None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:11.767992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f19fb0fc-fbf2-43c3-afe9-d60de4d300cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:11.858013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 825/1666] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:11.872014 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafd9637-c71f-447d-b103-27028799d4a6 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:11.872798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafd9637-c71f-447d-b103-27028799d4a6 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:11.872996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafd9637-c71f-447d-b103-27028799d4a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:11.889004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f8323-dd7d-48ae-bed0-97726530adf9 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:11.891805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f8323-dd7d-48ae-bed0-97726530adf9 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:11.891805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f8323-dd7d-48ae-bed0-97726530adf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:11.899564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:11.958187 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a89f8323-dd7d-48ae-bed0-97726530adf9 None None] MFA Rules not processed for user `c4a002047e3e42338e4f57ba49e877fb`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:11.960166 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a89f8323-dd7d-48ae-bed0-97726530adf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:11.994718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-dafd9637-c71f-447d-b103-27028799d4a6 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:12.001014 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-dafd9637-c71f-447d-b103-27028799d4a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:12.038529 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 842/1667] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:12.041606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafd9637-c71f-447d-b103-27028799d4a6 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:12.046712 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-dafd9637-c71f-447d-b103-27028799d4a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:12.090983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafd9637-c71f-447d-b103-27028799d4a6 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:12.102884 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:12.103607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:12.108956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:12.111699 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] PATH_INFO: `/v3/projects/9fb1fc3901ef47cc81c384173376f0d5` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:12.114030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9fb1fc3901ef47cc81c384173376f0d5)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:12.118552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:12.132969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 826/1668] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:11 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:12.132969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89f8323-dd7d-48ae-bed0-97726530adf9 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:12.132969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a89f8323-dd7d-48ae-bed0-97726530adf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:12.162907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89f8323-dd7d-48ae-bed0-97726530adf9 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:12.170074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d324277-f8b5-478d-8d6d-5830d8d61bec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:12.176035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d324277-f8b5-478d-8d6d-5830d8d61bec admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:12.176035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d324277-f8b5-478d-8d6d-5830d8d61bec admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:12.176035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d324277-f8b5-478d-8d6d-5830d8d61bec admin admin] PATH_INFO: `/v3/projects/a6e86637a7694247b50ef535d2a4984d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:12.176402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d324277-f8b5-478d-8d6d-5830d8d61bec admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6e86637a7694247b50ef535d2a4984d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:12.181641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d324277-f8b5-478d-8d6d-5830d8d61bec admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:12.298138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:12.329108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9fb1fc3901ef47cc81c384173376f0d5', 'request_id': 'req-dd24d625-5a82-4b82-933c-d3b5b68e7b04'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:12.345620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9fb1fc3901ef47cc81c384173376f0d5', 'request_id': 'req-dd24d625-5a82-4b82-933c-d3b5b68e7b04'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:12.355052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2d324277-f8b5-478d-8d6d-5830d8d61bec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6e86637a7694247b50ef535d2a4984d', 'request_id': 'req-2d324277-f8b5-478d-8d6d-5830d8d61bec'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:12.363735 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9fb1fc3901ef47cc81c384173376f0d5', 'request_id': 'req-dd24d625-5a82-4b82-933c-d3b5b68e7b04'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:12.378501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2d324277-f8b5-478d-8d6d-5830d8d61bec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6e86637a7694247b50ef535d2a4984d', 'request_id': 'req-2d324277-f8b5-478d-8d6d-5830d8d61bec'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:12.405240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2d324277-f8b5-478d-8d6d-5830d8d61bec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6e86637a7694247b50ef535d2a4984d', 'request_id': 'req-2d324277-f8b5-478d-8d6d-5830d8d61bec'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:12.408550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] The token cache is being invalidate because project 9fb1fc3901ef47cc81c384173376f0d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:12.409674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:12.410882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:12.419735 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 843/1669] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:12 2022] DELETE /identity/v3/projects/9fb1fc3901ef47cc81c384173376f0d5 => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:12.420773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:12.423695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:12.445145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2d324277-f8b5-478d-8d6d-5830d8d61bec admin admin] The token cache is being invalidate because project a6e86637a7694247b50ef535d2a4984d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:12.445451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2d324277-f8b5-478d-8d6d-5830d8d61bec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:12.452124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2d324277-f8b5-478d-8d6d-5830d8d61bec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:12.458931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 827/1670] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:12 2022] DELETE /identity/v3/projects/a6e86637a7694247b50ef535d2a4984d => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:12.469048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6cebdb-766b-418c-b591-17caf7c152df None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:12.469398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6cebdb-766b-418c-b591-17caf7c152df None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:12.469731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6cebdb-766b-418c-b591-17caf7c152df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:12.540462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-3f6cebdb-766b-418c-b591-17caf7c152df None None] MFA Rules not processed for user `c4a002047e3e42338e4f57ba49e877fb`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:12.542364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-3f6cebdb-766b-418c-b591-17caf7c152df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:12.560590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd24d625-5a82-4b82-933c-d3b5b68e7b04 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:12.665940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 828/1671] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:12 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:12.667651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a388f42-3f0e-4529-ab79-5648bb683f24 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:12.668143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a388f42-3f0e-4529-ab79-5648bb683f24 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:12.668488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a388f42-3f0e-4529-ab79-5648bb683f24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:12.674162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9d491c-ed79-4998-836f-d3f9255f28c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:12.675912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9d491c-ed79-4998-836f-d3f9255f28c4 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:12.676372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9d491c-ed79-4998-836f-d3f9255f28c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:12.676823 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9d491c-ed79-4998-836f-d3f9255f28c4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:12.678235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9d491c-ed79-4998-836f-d3f9255f28c4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:08:12.678811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9d491c-ed79-4998-836f-d3f9255f28c4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:12.681879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9d491c-ed79-4998-836f-d3f9255f28c4 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:12.698581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-0a388f42-3f0e-4529-ab79-5648bb683f24 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:12.700665 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0a388f42-3f0e-4529-ab79-5648bb683f24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:12.708809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 844/1672] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:08:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:08:12.716868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641ab597-40a7-4b84-bc15-37010609db7b None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:12.717375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641ab597-40a7-4b84-bc15-37010609db7b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:12.717789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 829/1673] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:12.719735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a388f42-3f0e-4529-ab79-5648bb683f24 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:12.721219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0a388f42-3f0e-4529-ab79-5648bb683f24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:12.732110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641ab597-40a7-4b84-bc15-37010609db7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:12.758830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:12.769171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-641ab597-40a7-4b84-bc15-37010609db7b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:12.771342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-641ab597-40a7-4b84-bc15-37010609db7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:12.787136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 845/1674] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:12.802093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641ab597-40a7-4b84-bc15-37010609db7b None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:12.805015 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-641ab597-40a7-4b84-bc15-37010609db7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:12.831891 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641ab597-40a7-4b84-bc15-37010609db7b None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:12.839291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd71a7a-4e01-47a8-9da1-7c8b57afca1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:12.840935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd71a7a-4e01-47a8-9da1-7c8b57afca1e admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:12.841199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd71a7a-4e01-47a8-9da1-7c8b57afca1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:12.841687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd71a7a-4e01-47a8-9da1-7c8b57afca1e admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:12.842935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd71a7a-4e01-47a8-9da1-7c8b57afca1e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:12.847543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd71a7a-4e01-47a8-9da1-7c8b57afca1e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:12.899042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a388f42-3f0e-4529-ab79-5648bb683f24 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:12.914457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5004bfd-606c-4073-872a-0c77620cd79c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:12.916535 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5004bfd-606c-4073-872a-0c77620cd79c service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:12.917447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5004bfd-606c-4073-872a-0c77620cd79c service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:12.918382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5004bfd-606c-4073-872a-0c77620cd79c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:12.921486 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b5004bfd-606c-4073-872a-0c77620cd79c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:12.929575 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 846/1675] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:12 2022] POST /identity/v3/projects => generated 371 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:12.933462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da9aba3-387b-433d-be48-cefb11c0d2ff None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:12.936197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da9aba3-387b-433d-be48-cefb11c0d2ff None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:12.938048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da9aba3-387b-433d-be48-cefb11c0d2ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:12.945506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5004bfd-606c-4073-872a-0c77620cd79c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:12.948336 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5004bfd-606c-4073-872a-0c77620cd79c service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:12.964002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 830/1676] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:12 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 246 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:12.965647 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa766d77-bdc8-4056-9398-36d890819743 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:12.966252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa766d77-bdc8-4056-9398-36d890819743 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:12.966684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa766d77-bdc8-4056-9398-36d890819743 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:13.003348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-aa766d77-bdc8-4056-9398-36d890819743 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:13.006252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-aa766d77-bdc8-4056-9398-36d890819743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:13.055088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-6da9aba3-387b-433d-be48-cefb11c0d2ff None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:13.062349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6da9aba3-387b-433d-be48-cefb11c0d2ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:13.230162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:13.290182 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 831/1677] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 325 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:13.290976 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa766d77-bdc8-4056-9398-36d890819743 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:13.316067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa766d77-bdc8-4056-9398-36d890819743 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:13.324773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994001d3-003a-4624-9e4d-490bfe788a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:13.326021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994001d3-003a-4624-9e4d-490bfe788a5d admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:13.326401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994001d3-003a-4624-9e4d-490bfe788a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:13.327804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994001d3-003a-4624-9e4d-490bfe788a5d admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:13.328713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994001d3-003a-4624-9e4d-490bfe788a5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:13.331708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994001d3-003a-4624-9e4d-490bfe788a5d admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:13.357855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 847/1678] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 426 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:13.361416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da9aba3-387b-433d-be48-cefb11c0d2ff None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:13.380402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 832/1679] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:13 2022] POST /identity/v3/users => generated 286 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:13.383385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994001d3-003a-4624-9e4d-490bfe788a5d admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:13.387782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da9aba3-387b-433d-be48-cefb11c0d2ff None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:13.399154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2ab77f-e205-4576-8c86-33ffd53c4689 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:13.400399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2ab77f-e205-4576-8c86-33ffd53c4689 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:13.400892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2ab77f-e205-4576-8c86-33ffd53c4689 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:13.401677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2ab77f-e205-4576-8c86-33ffd53c4689 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:13.403133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2ab77f-e205-4576-8c86-33ffd53c4689 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:08:13.403946 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2ab77f-e205-4576-8c86-33ffd53c4689 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:13.406396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994001d3-003a-4624-9e4d-490bfe788a5d admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:13.409783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2ab77f-e205-4576-8c86-33ffd53c4689 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:13.416666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de93ec9-1245-4ca0-8b26-4f6140328a66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:13.419018 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de93ec9-1245-4ca0-8b26-4f6140328a66 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:13.420630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de93ec9-1245-4ca0-8b26-4f6140328a66 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:13.421035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de93ec9-1245-4ca0-8b26-4f6140328a66 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:13.423545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de93ec9-1245-4ca0-8b26-4f6140328a66 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:08:13.423979 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de93ec9-1245-4ca0-8b26-4f6140328a66 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:13.429868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de93ec9-1245-4ca0-8b26-4f6140328a66 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:13.441773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 848/1680] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:08:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:08:13.455401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2ab77f-e205-4576-8c86-33ffd53c4689 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:13.482252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 833/1681] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:08:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:08:13.483451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de93ec9-1245-4ca0-8b26-4f6140328a66 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:13.501052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2ab77f-e205-4576-8c86-33ffd53c4689 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:13.511031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de93ec9-1245-4ca0-8b26-4f6140328a66 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:13.528855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942cd467-2681-4f18-8dcf-a0d6fad33da6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:13.531059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942cd467-2681-4f18-8dcf-a0d6fad33da6 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:13.531852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942cd467-2681-4f18-8dcf-a0d6fad33da6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:13.533224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942cd467-2681-4f18-8dcf-a0d6fad33da6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:13.534094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af863107-0dbf-4f07-a5a5-ec88cf160897 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:13.535018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942cd467-2681-4f18-8dcf-a0d6fad33da6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:13.535819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af863107-0dbf-4f07-a5a5-ec88cf160897 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:13.537687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942cd467-2681-4f18-8dcf-a0d6fad33da6 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:13.538158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af863107-0dbf-4f07-a5a5-ec88cf160897 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:13.539346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af863107-0dbf-4f07-a5a5-ec88cf160897 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:13.540290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af863107-0dbf-4f07-a5a5-ec88cf160897 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:13.544510 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af863107-0dbf-4f07-a5a5-ec88cf160897 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:13.571513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 849/1682] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:13 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:13.574689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942cd467-2681-4f18-8dcf-a0d6fad33da6 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:13.605547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942cd467-2681-4f18-8dcf-a0d6fad33da6 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:13.622824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 834/1683] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:13 2022] POST /identity/v3/projects => generated 377 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:13.622824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af863107-0dbf-4f07-a5a5-ec88cf160897 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:13.626305 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9593c6eb-c3aa-4346-abe6-8ff3b0aba418 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:13.627131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9593c6eb-c3aa-4346-abe6-8ff3b0aba418 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:13.631547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9593c6eb-c3aa-4346-abe6-8ff3b0aba418 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:13.632859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9593c6eb-c3aa-4346-abe6-8ff3b0aba418 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:13.634918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9593c6eb-c3aa-4346-abe6-8ff3b0aba418 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:13.638547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9593c6eb-c3aa-4346-abe6-8ff3b0aba418 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:13.707869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 850/1684] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:13 2022] POST /identity/v3/projects => generated 367 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:13.707869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9593c6eb-c3aa-4346-abe6-8ff3b0aba418 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:13.764468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:13.797659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af863107-0dbf-4f07-a5a5-ec88cf160897 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:13.890167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9593c6eb-c3aa-4346-abe6-8ff3b0aba418 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:13.905609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdafd621-fb0c-470b-877c-0c93e393df59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:13.906672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdafd621-fb0c-470b-877c-0c93e393df59 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:13.907120 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdafd621-fb0c-470b-877c-0c93e393df59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:13.907585 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdafd621-fb0c-470b-877c-0c93e393df59 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:13.908439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdafd621-fb0c-470b-877c-0c93e393df59 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:13.914142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdafd621-fb0c-470b-877c-0c93e393df59 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:13.961604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463855f6-2da1-4b2f-9884-b0c52c79e499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:13.962696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463855f6-2da1-4b2f-9884-b0c52c79e499 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:13.964398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463855f6-2da1-4b2f-9884-b0c52c79e499 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:13.965057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463855f6-2da1-4b2f-9884-b0c52c79e499 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:13.965516 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 851/1685] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:13 2022] POST /identity/v3/users => generated 289 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:13.966865 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdafd621-fb0c-470b-877c-0c93e393df59 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:13.974591 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463855f6-2da1-4b2f-9884-b0c52c79e499 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:13.995801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463855f6-2da1-4b2f-9884-b0c52c79e499 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:13.996773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdafd621-fb0c-470b-877c-0c93e393df59 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:14.009576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef62e9d-80b3-4e7e-ac54-5f7ced53a711 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:14.010220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef62e9d-80b3-4e7e-ac54-5f7ced53a711 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.010496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef62e9d-80b3-4e7e-ac54-5f7ced53a711 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.010772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef62e9d-80b3-4e7e-ac54-5f7ced53a711 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.011624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef62e9d-80b3-4e7e-ac54-5f7ced53a711 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:14.013520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef62e9d-80b3-4e7e-ac54-5f7ced53a711 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:14.053775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 835/1686] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:13 2022] GET /identity/v3/roles => generated 1420 bytes in 435 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:14.055026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463855f6-2da1-4b2f-9884-b0c52c79e499 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:14.072073 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 852/1687] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:13 2022] POST /identity/v3/users => generated 284 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:14.077997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef62e9d-80b3-4e7e-ac54-5f7ced53a711 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:14.093052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463855f6-2da1-4b2f-9884-b0c52c79e499 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:14.106973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef62e9d-80b3-4e7e-ac54-5f7ced53a711 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:14.117778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bf91d1-3ac1-428d-83ad-8571189cd56b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:14.118667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bf91d1-3ac1-428d-83ad-8571189cd56b admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.118961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bf91d1-3ac1-428d-83ad-8571189cd56b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.119232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bf91d1-3ac1-428d-83ad-8571189cd56b admin admin] PATH_INFO: `/v3/projects/1ef4c28d3fc94ed6960965e1f3f2d2b8/users/6d800f7c273f4c9f8874f84231e8f696/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.124206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3a576-924f-43bb-8e34-67925b869a89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:14.124909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3a576-924f-43bb-8e34-67925b869a89 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.125193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3a576-924f-43bb-8e34-67925b869a89 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.125472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3a576-924f-43bb-8e34-67925b869a89 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.126391 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce3a576-924f-43bb-8e34-67925b869a89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:14.128734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce3a576-924f-43bb-8e34-67925b869a89 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:14.166889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bf91d1-3ac1-428d-83ad-8571189cd56b admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ef4c28d3fc94ed6960965e1f3f2d2b8, user_id=6d800f7c273f4c9f8874f84231e8f696, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:14.167171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 836/1688] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:14 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:14.167171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3a576-924f-43bb-8e34-67925b869a89 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:14.172549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bf91d1-3ac1-428d-83ad-8571189cd56b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:14.192243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3a576-924f-43bb-8e34-67925b869a89 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:14.206572 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 853/1689] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:14 2022] PUT /identity/v3/projects/1ef4c28d3fc94ed6960965e1f3f2d2b8/users/6d800f7c273f4c9f8874f84231e8f696/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:14.217996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bf91d1-3ac1-428d-83ad-8571189cd56b admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:14.224677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de41c298-6b29-4641-91dc-2feb1c5dc452 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:14.225567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41c298-6b29-4641-91dc-2feb1c5dc452 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.225722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41c298-6b29-4641-91dc-2feb1c5dc452 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.225875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41c298-6b29-4641-91dc-2feb1c5dc452 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.226572 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de41c298-6b29-4641-91dc-2feb1c5dc452 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:14.230964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de41c298-6b29-4641-91dc-2feb1c5dc452 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:14.264411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 837/1690] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:14 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:14.265936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3759ae-21dd-4442-a12c-1d1c229a7756 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.266226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3759ae-21dd-4442-a12c-1d1c229a7756 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.266505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3759ae-21dd-4442-a12c-1d1c229a7756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.323693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:14.345505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-aa3759ae-21dd-4442-a12c-1d1c229a7756 None None] MFA Rules not processed for user `6d800f7c273f4c9f8874f84231e8f696`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:14.347338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-aa3759ae-21dd-4442-a12c-1d1c229a7756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:14.379997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bf91d1-3ac1-428d-83ad-8571189cd56b admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:14.396150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f6aa91-d8eb-4ef1-830b-de651818ed68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:14.397064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f6aa91-d8eb-4ef1-830b-de651818ed68 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.398009 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f6aa91-d8eb-4ef1-830b-de651818ed68 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.398009 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f6aa91-d8eb-4ef1-830b-de651818ed68 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.401278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f6aa91-d8eb-4ef1-830b-de651818ed68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:14.410606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f6aa91-d8eb-4ef1-830b-de651818ed68 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:14.451544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 854/1691] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:14 2022] GET /identity/v3/roles => generated 1420 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:14.453856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f6aa91-d8eb-4ef1-830b-de651818ed68 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:14.486718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f6aa91-d8eb-4ef1-830b-de651818ed68 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:14.507262 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bfbdc0-f09f-493f-ba21-e464fca6dbf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:14.508405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bfbdc0-f09f-493f-ba21-e464fca6dbf7 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.509313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bfbdc0-f09f-493f-ba21-e464fca6dbf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.509775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bfbdc0-f09f-493f-ba21-e464fca6dbf7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.517528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bfbdc0-f09f-493f-ba21-e464fca6dbf7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:14.520264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 838/1692] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:14 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:14.523395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bfbdc0-f09f-493f-ba21-e464fca6dbf7 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:14.525656 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3759ae-21dd-4442-a12c-1d1c229a7756 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:14.548812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3759ae-21dd-4442-a12c-1d1c229a7756 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:14.564330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592701ef-ec40-4de9-873c-98652cde97a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:14.565081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592701ef-ec40-4de9-873c-98652cde97a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.565367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592701ef-ec40-4de9-873c-98652cde97a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.565704 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592701ef-ec40-4de9-873c-98652cde97a1 admin admin] PATH_INFO: `/v3/projects/c57d09c6a1c84fd492393aebb2101715/users/79b46b24aade43c5b0c3821de4cfd3fe/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.587967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 855/1693] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:14 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:14.598674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bfbdc0-f09f-493f-ba21-e464fca6dbf7 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:14.604775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592701ef-ec40-4de9-873c-98652cde97a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c57d09c6a1c84fd492393aebb2101715, user_id=79b46b24aade43c5b0c3821de4cfd3fe, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:14.608831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592701ef-ec40-4de9-873c-98652cde97a1 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:14.639306 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 839/1694] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:14 2022] PUT /identity/v3/projects/c57d09c6a1c84fd492393aebb2101715/users/79b46b24aade43c5b0c3821de4cfd3fe/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:14.640404 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592701ef-ec40-4de9-873c-98652cde97a1 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:14.662584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bfbdc0-f09f-493f-ba21-e464fca6dbf7 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:14.733168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:14.802577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592701ef-ec40-4de9-873c-98652cde97a1 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:14.820554 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c9c0c8-1123-4e4a-822a-00261db6ca23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:14.821261 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c9c0c8-1123-4e4a-822a-00261db6ca23 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.821551 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c9c0c8-1123-4e4a-822a-00261db6ca23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.821836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c9c0c8-1123-4e4a-822a-00261db6ca23 admin admin] PATH_INFO: `/v3/projects/4449fa3a4fcb4f9598612b3ff3a54cbc/users/6cfac427a3bb4c93baf7d4e642d53f5f/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.824537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569475e0-6259-43b1-9f66-09a0bc5b848b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:14.825438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569475e0-6259-43b1-9f66-09a0bc5b848b admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.825874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569475e0-6259-43b1-9f66-09a0bc5b848b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.826308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569475e0-6259-43b1-9f66-09a0bc5b848b admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.827168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569475e0-6259-43b1-9f66-09a0bc5b848b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:14.834485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569475e0-6259-43b1-9f66-09a0bc5b848b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:14.857307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c9c0c8-1123-4e4a-822a-00261db6ca23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4449fa3a4fcb4f9598612b3ff3a54cbc, user_id=6cfac427a3bb4c93baf7d4e642d53f5f, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:14.863228 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c9c0c8-1123-4e4a-822a-00261db6ca23 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:14.905347 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 840/1695] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:14 2022] PUT /identity/v3/projects/4449fa3a4fcb4f9598612b3ff3a54cbc/users/6cfac427a3bb4c93baf7d4e642d53f5f/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:14.906900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa533f7e-00c5-4645-bd95-5fcd74231512 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.907192 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa533f7e-00c5-4645-bd95-5fcd74231512 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.907500 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa533f7e-00c5-4645-bd95-5fcd74231512 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.909723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 856/1696] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:14 2022] POST /identity/v3/projects => generated 373 bytes in 319 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:14.911343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f5b5c5-410a-487c-8bdf-1407245250fe None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.915982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f5b5c5-410a-487c-8bdf-1407245250fe None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.916422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f5b5c5-410a-487c-8bdf-1407245250fe None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.923619 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 857/1697] 10.222.0.42 () {54 vars in 912 bytes} [Sun Jul 31 01:08:14 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:08:14.925097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dae07b-85bf-4846-baa5-2adb38546166 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:14.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dae07b-85bf-4846-baa5-2adb38546166 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:14.928020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dae07b-85bf-4846-baa5-2adb38546166 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:14.986927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-aa533f7e-00c5-4645-bd95-5fcd74231512 None None] MFA Rules not processed for user `79b46b24aade43c5b0c3821de4cfd3fe`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:14.988239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-aa533f7e-00c5-4645-bd95-5fcd74231512 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:15.048821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-68dae07b-85bf-4846-baa5-2adb38546166 None None] MFA Rules not processed for user `6cfac427a3bb4c93baf7d4e642d53f5f`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:15.051503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-68dae07b-85bf-4846-baa5-2adb38546166 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:15.236403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 841/1698] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:14 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 324 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:15.239865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa533f7e-00c5-4645-bd95-5fcd74231512 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:15.378761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 858/1699] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:14 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 450 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:15.378761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4b5e2e-73f0-4d46-a1c7-edfad112ae34 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:15.378761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4b5e2e-73f0-4d46-a1c7-edfad112ae34 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:15.378761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4b5e2e-73f0-4d46-a1c7-edfad112ae34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:15.453431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa533f7e-00c5-4645-bd95-5fcd74231512 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:15.472264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef4683a-444a-41bb-8569-82cb8abf4a0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:15.473378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef4683a-444a-41bb-8569-82cb8abf4a0f admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:15.473378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef4683a-444a-41bb-8569-82cb8abf4a0f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:15.473681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef4683a-444a-41bb-8569-82cb8abf4a0f admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:15.474120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef4683a-444a-41bb-8569-82cb8abf4a0f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:15.476150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef4683a-444a-41bb-8569-82cb8abf4a0f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:15.508541 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ab4b5e2e-73f0-4d46-a1c7-edfad112ae34 None None] MFA Rules not processed for user `94e868131d33472b9ded8486274a07b9`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:15.512957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ab4b5e2e-73f0-4d46-a1c7-edfad112ae34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:15.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 842/1700] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:15 2022] POST /identity/v3/users => generated 293 bytes in 299 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:15.534084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef4683a-444a-41bb-8569-82cb8abf4a0f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:15.555024 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef4683a-444a-41bb-8569-82cb8abf4a0f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:15.563235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:15.571241 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec6c5c4-e23b-4b6f-b3ce-5a1cc0065e3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:15.571925 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec6c5c4-e23b-4b6f-b3ce-5a1cc0065e3e admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:15.572535 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec6c5c4-e23b-4b6f-b3ce-5a1cc0065e3e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:15.572661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec6c5c4-e23b-4b6f-b3ce-5a1cc0065e3e admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:15.573410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec6c5c4-e23b-4b6f-b3ce-5a1cc0065e3e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:15.576371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec6c5c4-e23b-4b6f-b3ce-5a1cc0065e3e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:15.660254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 843/1701] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:15 2022] POST /identity/v3/projects => generated 377 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:15.660254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec6c5c4-e23b-4b6f-b3ce-5a1cc0065e3e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:15.670970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 859/1702] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:08:15 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:15.689424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4b5e2e-73f0-4d46-a1c7-edfad112ae34 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:15.709559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:15.730736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4b5e2e-73f0-4d46-a1c7-edfad112ae34 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:15.951721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e023d3b-fdb6-4528-aae8-34ca18aca61b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:15.953017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e023d3b-fdb6-4528-aae8-34ca18aca61b admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:15.953449 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e023d3b-fdb6-4528-aae8-34ca18aca61b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:15.953604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e023d3b-fdb6-4528-aae8-34ca18aca61b admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:15.954308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e023d3b-fdb6-4528-aae8-34ca18aca61b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:15.956050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e023d3b-fdb6-4528-aae8-34ca18aca61b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:15.966762 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec6c5c4-e23b-4b6f-b3ce-5a1cc0065e3e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:15.981346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f847e494-e782-4d48-bb65-cb29a5b09fea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:15.984659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f847e494-e782-4d48-bb65-cb29a5b09fea admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:15.985419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f847e494-e782-4d48-bb65-cb29a5b09fea admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:15.986757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f847e494-e782-4d48-bb65-cb29a5b09fea admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:15.989718 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f847e494-e782-4d48-bb65-cb29a5b09fea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:15.991100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 860/1703] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:15 2022] GET /identity/v3/roles => generated 1420 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:15.991676 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e023d3b-fdb6-4528-aae8-34ca18aca61b admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:15.999540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f847e494-e782-4d48-bb65-cb29a5b09fea admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:16.006575 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e023d3b-fdb6-4528-aae8-34ca18aca61b admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:16.015845 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b63059-9625-4155-9eb2-590d27eb7000 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:16.016485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b63059-9625-4155-9eb2-590d27eb7000 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:16.016765 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b63059-9625-4155-9eb2-590d27eb7000 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:16.017039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b63059-9625-4155-9eb2-590d27eb7000 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:16.017724 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b63059-9625-4155-9eb2-590d27eb7000 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:16.019971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b63059-9625-4155-9eb2-590d27eb7000 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:16.065450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 861/1704] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:15 2022] POST /identity/v3/users => generated 295 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:16.066700 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b63059-9625-4155-9eb2-590d27eb7000 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:16.068806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-37b63059-9625-4155-9eb2-590d27eb7000 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:16.111201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 844/1705] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:15 2022] POST /identity/v3/projects => generated 369 bytes in 457 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:16.114164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f847e494-e782-4d48-bb65-cb29a5b09fea admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:16.167156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b63059-9625-4155-9eb2-590d27eb7000 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:16.241393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffdb94a-9ea8-4643-ba15-7c19f6f39191 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:16.242037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffdb94a-9ea8-4643-ba15-7c19f6f39191 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:16.242421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffdb94a-9ea8-4643-ba15-7c19f6f39191 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:16.242803 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffdb94a-9ea8-4643-ba15-7c19f6f39191 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:16.252336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffdb94a-9ea8-4643-ba15-7c19f6f39191 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:16.253882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffdb94a-9ea8-4643-ba15-7c19f6f39191 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:16.272097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 862/1706] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:16 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:16.273088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffdb94a-9ea8-4643-ba15-7c19f6f39191 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:16.292045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:16.348171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f847e494-e782-4d48-bb65-cb29a5b09fea admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:16.368613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c29761a-e7e2-4136-8786-25637150d395 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:16.371576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c29761a-e7e2-4136-8786-25637150d395 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:16.371576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c29761a-e7e2-4136-8786-25637150d395 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:16.371576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c29761a-e7e2-4136-8786-25637150d395 admin admin] PATH_INFO: `/v3/projects/5422d71676324b2e930a9e8d5f3a71cb/users/b4b8449f572b40d3a70324fe9f711a08/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:16.405259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c29761a-e7e2-4136-8786-25637150d395 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5422d71676324b2e930a9e8d5f3a71cb, user_id=b4b8449f572b40d3a70324fe9f711a08, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:16.412888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffdb94a-9ea8-4643-ba15-7c19f6f39191 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:16.414813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c29761a-e7e2-4136-8786-25637150d395 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:16.426935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4126ece-79bc-42a5-81dc-e7cb49f423e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:16.427825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4126ece-79bc-42a5-81dc-e7cb49f423e7 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:16.428281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4126ece-79bc-42a5-81dc-e7cb49f423e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:16.428567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4126ece-79bc-42a5-81dc-e7cb49f423e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:16.432912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4126ece-79bc-42a5-81dc-e7cb49f423e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:16.436580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4126ece-79bc-42a5-81dc-e7cb49f423e7 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:16.474186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 863/1707] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:16 2022] GET /identity/v3/roles => generated 1420 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:16.477440 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4126ece-79bc-42a5-81dc-e7cb49f423e7 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:16.494634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 845/1708] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:16 2022] PUT /identity/v3/projects/5422d71676324b2e930a9e8d5f3a71cb/users/b4b8449f572b40d3a70324fe9f711a08/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:16.495452 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c29761a-e7e2-4136-8786-25637150d395 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:16.516318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4126ece-79bc-42a5-81dc-e7cb49f423e7 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:16.777093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c29761a-e7e2-4136-8786-25637150d395 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:16.778320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2001ba98-9707-425d-801c-3e583e0d2d6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:16.780312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2001ba98-9707-425d-801c-3e583e0d2d6a admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:16.780694 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2001ba98-9707-425d-801c-3e583e0d2d6a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:16.785792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2001ba98-9707-425d-801c-3e583e0d2d6a admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:16.789335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2001ba98-9707-425d-801c-3e583e0d2d6a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:16.798668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2001ba98-9707-425d-801c-3e583e0d2d6a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:16.800998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65e39a5-74d4-402e-8ec3-929e17a69417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:16.806893 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65e39a5-74d4-402e-8ec3-929e17a69417 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:16.807186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65e39a5-74d4-402e-8ec3-929e17a69417 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:16.807599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65e39a5-74d4-402e-8ec3-929e17a69417 admin admin] PATH_INFO: `/v3/projects/353d268d38764c48a252cdf9a5146156/users/e6d9332df3fc4b3a8d2b7b8a738521db/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:16.886134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65e39a5-74d4-402e-8ec3-929e17a69417 admin admin] RBAC: Authorizing `identity:create_grant(project_id=353d268d38764c48a252cdf9a5146156, user_id=e6d9332df3fc4b3a8d2b7b8a738521db, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:16.893708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 864/1709] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:16 2022] POST /identity/v3/users => generated 291 bytes in 416 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:16.893708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2001ba98-9707-425d-801c-3e583e0d2d6a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:16.900709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65e39a5-74d4-402e-8ec3-929e17a69417 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:16.946299 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 846/1710] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:16 2022] PUT /identity/v3/projects/353d268d38764c48a252cdf9a5146156/users/e6d9332df3fc4b3a8d2b7b8a738521db/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:16.947052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65e39a5-74d4-402e-8ec3-929e17a69417 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:16.949793 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2001ba98-9707-425d-801c-3e583e0d2d6a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:16.975586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:17.167179 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b60ad51-3ce1-4ee9-b873-20ad6933fb35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:17.168087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b60ad51-3ce1-4ee9-b873-20ad6933fb35 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:17.168405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b60ad51-3ce1-4ee9-b873-20ad6933fb35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:17.168715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b60ad51-3ce1-4ee9-b873-20ad6933fb35 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:17.170109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b60ad51-3ce1-4ee9-b873-20ad6933fb35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:17.172233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b60ad51-3ce1-4ee9-b873-20ad6933fb35 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:17.177396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65e39a5-74d4-402e-8ec3-929e17a69417 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:17.197976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 865/1711] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:16 2022] GET /identity/v3/roles => generated 1420 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:17.199069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b60ad51-3ce1-4ee9-b873-20ad6933fb35 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:17.208173 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463c88a3-067f-41a2-a61b-88455de7a559 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:17.208830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463c88a3-067f-41a2-a61b-88455de7a559 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:17.209108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463c88a3-067f-41a2-a61b-88455de7a559 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:17.209383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463c88a3-067f-41a2-a61b-88455de7a559 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:17.210444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463c88a3-067f-41a2-a61b-88455de7a559 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:17.211889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463c88a3-067f-41a2-a61b-88455de7a559 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:17.232197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b60ad51-3ce1-4ee9-b873-20ad6933fb35 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:17.255186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 847/1712] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:16 2022] GET /identity/v3/roles => generated 1420 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:17.255186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463c88a3-067f-41a2-a61b-88455de7a559 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:17.257879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d12b2c7-4e26-4f72-866b-54d9af0c722e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:17.260453 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d12b2c7-4e26-4f72-866b-54d9af0c722e admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:17.261253 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d12b2c7-4e26-4f72-866b-54d9af0c722e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:17.262910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d12b2c7-4e26-4f72-866b-54d9af0c722e admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:17.265548 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d12b2c7-4e26-4f72-866b-54d9af0c722e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:17.275575 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d12b2c7-4e26-4f72-866b-54d9af0c722e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:17.308118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463c88a3-067f-41a2-a61b-88455de7a559 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:17.337094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1050d49a-91d7-4f49-b41a-a1877805cd10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:17.341508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1050d49a-91d7-4f49-b41a-a1877805cd10 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:17.342279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1050d49a-91d7-4f49-b41a-a1877805cd10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:17.342774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1050d49a-91d7-4f49-b41a-a1877805cd10 admin admin] PATH_INFO: `/v3/projects/5422d71676324b2e930a9e8d5f3a71cb/users/b4b8449f572b40d3a70324fe9f711a08/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:17.344921 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 866/1713] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:17 2022] GET /identity/v3/roles => generated 1420 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:17.346854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d12b2c7-4e26-4f72-866b-54d9af0c722e admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:17.354731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1050d49a-91d7-4f49-b41a-a1877805cd10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5422d71676324b2e930a9e8d5f3a71cb, user_id=b4b8449f572b40d3a70324fe9f711a08, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:17.367664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1050d49a-91d7-4f49-b41a-a1877805cd10 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:17.390203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d12b2c7-4e26-4f72-866b-54d9af0c722e admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:17.416841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 848/1714] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:17 2022] PUT /identity/v3/projects/5422d71676324b2e930a9e8d5f3a71cb/users/b4b8449f572b40d3a70324fe9f711a08/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:17.418595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1050d49a-91d7-4f49-b41a-a1877805cd10 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:17.460977 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:17.596577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1050d49a-91d7-4f49-b41a-a1877805cd10 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:17.608899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ff9e4a-fb99-4b78-aafc-034c41fbc71f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:17.609266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ff9e4a-fb99-4b78-aafc-034c41fbc71f admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:17.619007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ff9e4a-fb99-4b78-aafc-034c41fbc71f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:17.620769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ff9e4a-fb99-4b78-aafc-034c41fbc71f admin admin] PATH_INFO: `/v3/projects/353d268d38764c48a252cdf9a5146156/users/e6d9332df3fc4b3a8d2b7b8a738521db/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:17.628475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ff9e4a-fb99-4b78-aafc-034c41fbc71f admin admin] RBAC: Authorizing `identity:create_grant(project_id=353d268d38764c48a252cdf9a5146156, user_id=e6d9332df3fc4b3a8d2b7b8a738521db, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:17.636899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ff9e4a-fb99-4b78-aafc-034c41fbc71f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:17.662479 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 849/1715] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:17 2022] PUT /identity/v3/projects/353d268d38764c48a252cdf9a5146156/users/e6d9332df3fc4b3a8d2b7b8a738521db/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:17.663472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0de7d27-b7bb-4856-9104-dfa097cc7649 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:17.663916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0de7d27-b7bb-4856-9104-dfa097cc7649 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:17.664078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0de7d27-b7bb-4856-9104-dfa097cc7649 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:17.684089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf011ebe-9540-46b6-9fec-8c5e581324de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:17.685603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf011ebe-9540-46b6-9fec-8c5e581324de admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:17.686116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf011ebe-9540-46b6-9fec-8c5e581324de admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:17.686601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf011ebe-9540-46b6-9fec-8c5e581324de admin admin] PATH_INFO: `/v3/projects/095b7ce8ef90456f8c9de94e187f94f8/users/1bf4dea1e8d349e4b19cc83b99e1423e/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:17.728988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf011ebe-9540-46b6-9fec-8c5e581324de admin admin] RBAC: Authorizing `identity:create_grant(project_id=095b7ce8ef90456f8c9de94e187f94f8, user_id=1bf4dea1e8d349e4b19cc83b99e1423e, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:17.730818 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-f0de7d27-b7bb-4856-9104-dfa097cc7649 None None] MFA Rules not processed for user `b4b8449f572b40d3a70324fe9f711a08`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:17.734605 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf011ebe-9540-46b6-9fec-8c5e581324de admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:17.734773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f0de7d27-b7bb-4856-9104-dfa097cc7649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:17.790811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 867/1716] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:17 2022] PUT /identity/v3/projects/095b7ce8ef90456f8c9de94e187f94f8/users/1bf4dea1e8d349e4b19cc83b99e1423e/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:17.793511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c4edad-9c8f-4a4a-a07c-08cbba58b358 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:17.793996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c4edad-9c8f-4a4a-a07c-08cbba58b358 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:17.794480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c4edad-9c8f-4a4a-a07c-08cbba58b358 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:17.853950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 850/1717] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:17 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:17.856473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65769f18-a053-4aad-b166-5dc810c09322 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:17.856768 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65769f18-a053-4aad-b166-5dc810c09322 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:17.857045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65769f18-a053-4aad-b166-5dc810c09322 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:17.885131 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:17.917183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-65769f18-a053-4aad-b166-5dc810c09322 None None] MFA Rules not processed for user `e6d9332df3fc4b3a8d2b7b8a738521db`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:17.919051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-65769f18-a053-4aad-b166-5dc810c09322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:17.946117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b6c4edad-9c8f-4a4a-a07c-08cbba58b358 None None] MFA Rules not processed for user `20df8d74c87143dd81dbaae76b3c2b62`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:17.947695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b6c4edad-9c8f-4a4a-a07c-08cbba58b358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:18.104982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 851/1718] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:17 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:18.107590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65769f18-a053-4aad-b166-5dc810c09322 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:18.231403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 868/1719] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:17 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 429 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:18.231403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c4edad-9c8f-4a4a-a07c-08cbba58b358 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:18.231403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b6c4edad-9c8f-4a4a-a07c-08cbba58b358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:18.397981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c4edad-9c8f-4a4a-a07c-08cbba58b358 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:18.408139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0ce7ca-61b4-4c81-b9ea-75e8f52d8fe1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:18.413728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0ce7ca-61b4-4c81-b9ea-75e8f52d8fe1 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:18.413728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0ce7ca-61b4-4c81-b9ea-75e8f52d8fe1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:18.414075 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0ce7ca-61b4-4c81-b9ea-75e8f52d8fe1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:18.442550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65769f18-a053-4aad-b166-5dc810c09322 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:18.453060 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2744e7-8f53-46df-b42c-2285559fc626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:18.454147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2744e7-8f53-46df-b42c-2285559fc626 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:18.454598 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2744e7-8f53-46df-b42c-2285559fc626 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:18.455032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2744e7-8f53-46df-b42c-2285559fc626 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:18.456541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2744e7-8f53-46df-b42c-2285559fc626 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:18.458965 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2744e7-8f53-46df-b42c-2285559fc626 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:18.479438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 852/1720] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:18 2022] GET /identity/v3/roles => generated 1420 bytes in 370 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:18.481016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2744e7-8f53-46df-b42c-2285559fc626 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:18.483000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9d2744e7-8f53-46df-b42c-2285559fc626 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:18.504406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2744e7-8f53-46df-b42c-2285559fc626 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:18.510947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2736dc8c-527e-40a7-9149-fba9f48a01d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:18.511628 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2736dc8c-527e-40a7-9149-fba9f48a01d1 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:18.512745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2736dc8c-527e-40a7-9149-fba9f48a01d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:18.512745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2736dc8c-527e-40a7-9149-fba9f48a01d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:18.567282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:18.602937 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0ce7ca-61b4-4c81-b9ea-75e8f52d8fe1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:18.604664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0ce7ca-61b4-4c81-b9ea-75e8f52d8fe1 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:18.621320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 869/1721] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:18 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:18.622222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0ce7ca-61b4-4c81-b9ea-75e8f52d8fe1 service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:18.771686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0ce7ca-61b4-4c81-b9ea-75e8f52d8fe1 service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:18.774281 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2736dc8c-527e-40a7-9149-fba9f48a01d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:18.778071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2736dc8c-527e-40a7-9149-fba9f48a01d1 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:18.779105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d97868-bfaa-4be5-911b-cd1ebdaa0fe0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:18.779923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d97868-bfaa-4be5-911b-cd1ebdaa0fe0 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:18.780336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d97868-bfaa-4be5-911b-cd1ebdaa0fe0 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:18.780893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d97868-bfaa-4be5-911b-cd1ebdaa0fe0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:18.803810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d97868-bfaa-4be5-911b-cd1ebdaa0fe0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:18.807793 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d97868-bfaa-4be5-911b-cd1ebdaa0fe0 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:18.837544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 853/1722] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:18 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:18.842761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 870/1723] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:18 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:18.844560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3157f99c-d601-4c81-b1fb-b4b34bac11be None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:18.845116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3157f99c-d601-4c81-b1fb-b4b34bac11be None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:18.845675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3157f99c-d601-4c81-b1fb-b4b34bac11be None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:18.845802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2736dc8c-527e-40a7-9149-fba9f48a01d1 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:18.853694 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 871/1724] 10.222.0.42 () {54 vars in 961 bytes} [Sun Jul 31 01:08:18 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:08:18.855235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f9cd4a-5091-4c8f-9bfc-ffd1271bbc82 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:18.855934 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f9cd4a-5091-4c8f-9bfc-ffd1271bbc82 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:18.856349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f9cd4a-5091-4c8f-9bfc-ffd1271bbc82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:18.882055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-16f9cd4a-5091-4c8f-9bfc-ffd1271bbc82 None None] MFA Rules not processed for user `1d562ccd50e4492281a9f435f809d49b`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:18.883639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-16f9cd4a-5091-4c8f-9bfc-ffd1271bbc82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:18.918352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2736dc8c-527e-40a7-9149-fba9f48a01d1 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:18.946563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a359e4d7-f66a-41d6-b683-2389164dd1b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:18.947962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a359e4d7-f66a-41d6-b683-2389164dd1b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:18.948441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a359e4d7-f66a-41d6-b683-2389164dd1b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:18.948918 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a359e4d7-f66a-41d6-b683-2389164dd1b7 admin admin] PATH_INFO: `/v3/projects/095b7ce8ef90456f8c9de94e187f94f8/users/1bf4dea1e8d349e4b19cc83b99e1423e/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:18.963682 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a359e4d7-f66a-41d6-b683-2389164dd1b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=095b7ce8ef90456f8c9de94e187f94f8, user_id=1bf4dea1e8d349e4b19cc83b99e1423e, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:18.989805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a359e4d7-f66a-41d6-b683-2389164dd1b7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:19.035980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 854/1725] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:18 2022] PUT /identity/v3/projects/095b7ce8ef90456f8c9de94e187f94f8/users/1bf4dea1e8d349e4b19cc83b99e1423e/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:19.052037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0689b58-a772-44d4-9b5d-6796379d65fa None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:19.052501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0689b58-a772-44d4-9b5d-6796379d65fa None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:19.052955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0689b58-a772-44d4-9b5d-6796379d65fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:19.055607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 872/1726] 10.222.0.42 () {58 vars in 1058 bytes} [Sun Jul 31 01:08:18 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:19.076654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f9cd4a-5091-4c8f-9bfc-ffd1271bbc82 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:19.090074 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f9cd4a-5091-4c8f-9bfc-ffd1271bbc82 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:19.150654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:19.175804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-f0689b58-a772-44d4-9b5d-6796379d65fa None None] MFA Rules not processed for user `1bf4dea1e8d349e4b19cc83b99e1423e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:19.177932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f0689b58-a772-44d4-9b5d-6796379d65fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:19.348233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7358083e-23b0-424a-b89f-ebddd5f55ba3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:19.349329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7358083e-23b0-424a-b89f-ebddd5f55ba3 service placement] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:19.350323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7358083e-23b0-424a-b89f-ebddd5f55ba3 service placement] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:19.350869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7358083e-23b0-424a-b89f-ebddd5f55ba3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:19.353188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7358083e-23b0-424a-b89f-ebddd5f55ba3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:19.369343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:19.374550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7358083e-23b0-424a-b89f-ebddd5f55ba3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:19.377048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7358083e-23b0-424a-b89f-ebddd5f55ba3 service placement] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:19.403088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 873/1727] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:19 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:19.418822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7358083e-23b0-424a-b89f-ebddd5f55ba3 service placement] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:19.500649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 855/1728] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:19 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 451 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:19.609822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7358083e-23b0-424a-b89f-ebddd5f55ba3 service placement] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:19.617894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0201e3d7-4e8d-4b54-a117-ef77826a31cc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:19.618782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0201e3d7-4e8d-4b54-a117-ef77826a31cc service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:19.618998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0201e3d7-4e8d-4b54-a117-ef77826a31cc service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:19.619843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0201e3d7-4e8d-4b54-a117-ef77826a31cc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:19.785076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0201e3d7-4e8d-4b54-a117-ef77826a31cc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:19.803467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0201e3d7-4e8d-4b54-a117-ef77826a31cc service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:19.835122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 874/1729] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:19 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:20.151234 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c33f0-f234-4a0f-9d0d-9e1365e98009 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:20.152041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c33f0-f234-4a0f-9d0d-9e1365e98009 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:20.152358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c33f0-f234-4a0f-9d0d-9e1365e98009 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:20.203687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0201e3d7-4e8d-4b54-a117-ef77826a31cc service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:20.244391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0201e3d7-4e8d-4b54-a117-ef77826a31cc service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:20.285059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7491828f-a829-41e2-9df1-2190d5a90267 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:20.286155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7491828f-a829-41e2-9df1-2190d5a90267 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:20.286648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7491828f-a829-41e2-9df1-2190d5a90267 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:20.287134 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7491828f-a829-41e2-9df1-2190d5a90267 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:20.318076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b24c33f0-f234-4a0f-9d0d-9e1365e98009 None None] MFA Rules not processed for user `56732ebe8ee24f3f9ce8f87746e8b161`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:20.326781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b24c33f0-f234-4a0f-9d0d-9e1365e98009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:20.335344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7491828f-a829-41e2-9df1-2190d5a90267 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:20.338231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7491828f-a829-41e2-9df1-2190d5a90267 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:20.366752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 875/1730] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:20 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:20.741170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 856/1731] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:20 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 593 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:20.783857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7491828f-a829-41e2-9df1-2190d5a90267 service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:20.786471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c33f0-f234-4a0f-9d0d-9e1365e98009 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:20.792418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b24c33f0-f234-4a0f-9d0d-9e1365e98009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:20.872743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7491828f-a829-41e2-9df1-2190d5a90267 service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:20.907460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1052fa72-b130-4c83-8ebb-f0c1e59a251c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:20.907460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1052fa72-b130-4c83-8ebb-f0c1e59a251c service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:20.907460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1052fa72-b130-4c83-8ebb-f0c1e59a251c service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:20.907460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1052fa72-b130-4c83-8ebb-f0c1e59a251c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:21.078278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1052fa72-b130-4c83-8ebb-f0c1e59a251c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:21.079842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1052fa72-b130-4c83-8ebb-f0c1e59a251c service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:21.101981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 876/1732] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:20 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:21.223817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c33f0-f234-4a0f-9d0d-9e1365e98009 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:21.266071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5f1ef9-7213-42cd-8836-e5a72350c81a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:21.266071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5f1ef9-7213-42cd-8836-e5a72350c81a service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:21.266071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5f1ef9-7213-42cd-8836-e5a72350c81a service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:21.266071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5f1ef9-7213-42cd-8836-e5a72350c81a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:21.304745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5f1ef9-7213-42cd-8836-e5a72350c81a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:21.306430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5f1ef9-7213-42cd-8836-e5a72350c81a service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:21.351149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 857/1733] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:20 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 566 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:23.516906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1166dc7e-66cd-4ad0-b2ae-98e49fd3b01e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:23.517479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1166dc7e-66cd-4ad0-b2ae-98e49fd3b01e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:23.518738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1166dc7e-66cd-4ad0-b2ae-98e49fd3b01e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:23.571203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-1166dc7e-66cd-4ad0-b2ae-98e49fd3b01e None None] MFA Rules not processed for user `20df8d74c87143dd81dbaae76b3c2b62`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:23.574354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-1166dc7e-66cd-4ad0-b2ae-98e49fd3b01e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:23.662154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:23.726912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 877/1734] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:23 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:24.863478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcc3343-59e9-4c18-8057-15e1871afb8b None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:24.864197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcc3343-59e9-4c18-8057-15e1871afb8b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:24.866056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcc3343-59e9-4c18-8057-15e1871afb8b None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:24.868739 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 858/1735] 10.222.0.42 () {54 vars in 912 bytes} [Sun Jul 31 01:08:24 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:08:24.878240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b135c6c6-3ce2-44a0-8143-b90edd7fc8bd None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:24.879283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b135c6c6-3ce2-44a0-8143-b90edd7fc8bd None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:24.879848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b135c6c6-3ce2-44a0-8143-b90edd7fc8bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:24.927713 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b135c6c6-3ce2-44a0-8143-b90edd7fc8bd None None] MFA Rules not processed for user `94e868131d33472b9ded8486274a07b9`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:24.929286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b135c6c6-3ce2-44a0-8143-b90edd7fc8bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:25.084980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 878/1736] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:08:24 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:25.086302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcc3343-59e9-4c18-8057-15e1871afb8b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:25.088345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-4bcc3343-59e9-4c18-8057-15e1871afb8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:25.120901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcc3343-59e9-4c18-8057-15e1871afb8b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:25.138523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce937bc1-edcc-425b-99b6-940ad1e5cf34 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:25.139636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce937bc1-edcc-425b-99b6-940ad1e5cf34 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:25.140136 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce937bc1-edcc-425b-99b6-940ad1e5cf34 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:25.140600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce937bc1-edcc-425b-99b6-940ad1e5cf34 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:25.161269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce937bc1-edcc-425b-99b6-940ad1e5cf34 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:25.164386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce937bc1-edcc-425b-99b6-940ad1e5cf34 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:25.182355 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:25.211289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 859/1737] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:25 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:25.805689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb369b35-4f0c-4878-9e69-9a6ded018437 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:25.806053 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb369b35-4f0c-4878-9e69-9a6ded018437 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:25.806340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb369b35-4f0c-4878-9e69-9a6ded018437 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:25.836631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-eb369b35-4f0c-4878-9e69-9a6ded018437 None None] MFA Rules not processed for user `56732ebe8ee24f3f9ce8f87746e8b161`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:25.837868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-eb369b35-4f0c-4878-9e69-9a6ded018437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:25.849631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 879/1738] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:25 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:27.864998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce937bc1-edcc-425b-99b6-940ad1e5cf34 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:28.067299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ecd69-bc17-46a4-8fd0-e265dc228665 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:28.068477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ecd69-bc17-46a4-8fd0-e265dc228665 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:28.068963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ecd69-bc17-46a4-8fd0-e265dc228665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:28.120195 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce937bc1-edcc-425b-99b6-940ad1e5cf34 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:28.144202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:28.145875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:28.146934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:28.147993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] PATH_INFO: `/v3/users/6cfac427a3bb4c93baf7d4e642d53f5f` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:28.153186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] RBAC: Authorizing `identity:delete_user(user_id=6cfac427a3bb4c93baf7d4e642d53f5f)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:28.160587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:28.237865 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-6a4ecd69-bc17-46a4-8fd0-e265dc228665 None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:28.240333 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6a4ecd69-bc17-46a4-8fd0-e265dc228665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:28.273734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6cfac427a3bb4c93baf7d4e642d53f5f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:28.288444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6cfac427a3bb4c93baf7d4e642d53f5f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:28.289848 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6cfac427a3bb4c93baf7d4e642d53f5f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:28.338909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6cfac427a3bb4c93baf7d4e642d53f5f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:28.341461 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6cfac427a3bb4c93baf7d4e642d53f5f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:28.377446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 860/1739] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:27 2022] DELETE /identity/v3/users/6cfac427a3bb4c93baf7d4e642d53f5f => generated 0 bytes in 514 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:28.379389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:28.384510 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:28.486657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 880/1740] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:08:28 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 421 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:28.488559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7f783-3e52-4bbe-a7dd-c8620f1d27b9 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:28.489257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7f783-3e52-4bbe-a7dd-c8620f1d27b9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:28.489739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7f783-3e52-4bbe-a7dd-c8620f1d27b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:28.550891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a65bc3c-cbbc-408d-8569-7f34044a27ea admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:28.575167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6179ab3a-8859-420d-80e8-421768bc183a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:28.584615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6179ab3a-8859-420d-80e8-421768bc183a service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:28.584615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6179ab3a-8859-420d-80e8-421768bc183a service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:28.584615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6179ab3a-8859-420d-80e8-421768bc183a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:28.597028 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6179ab3a-8859-420d-80e8-421768bc183a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:28.626233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b7b7f783-3e52-4bbe-a7dd-c8620f1d27b9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:28.629339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b7b7f783-3e52-4bbe-a7dd-c8620f1d27b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:28.687571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:28.760620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6179ab3a-8859-420d-80e8-421768bc183a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:28.762687 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6179ab3a-8859-420d-80e8-421768bc183a service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:28.792464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:28.796901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 861/1741] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:28 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:28.838127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 881/1742] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 350 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:28.850053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6179ab3a-8859-420d-80e8-421768bc183a service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:28.853121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6179ab3a-8859-420d-80e8-421768bc183a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:28.883846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b7f783-3e52-4bbe-a7dd-c8620f1d27b9 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:28.885416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b7b7f783-3e52-4bbe-a7dd-c8620f1d27b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:29.075573 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6179ab3a-8859-420d-80e8-421768bc183a service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:29.088199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3887d03-a2f5-4c7b-afef-5b549de675cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:29.089170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3887d03-a2f5-4c7b-afef-5b549de675cd service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:29.091707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3887d03-a2f5-4c7b-afef-5b549de675cd service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:29.092660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3887d03-a2f5-4c7b-afef-5b549de675cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:29.121059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3887d03-a2f5-4c7b-afef-5b549de675cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:29.128672 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3887d03-a2f5-4c7b-afef-5b549de675cd service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:29.170601 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 862/1743] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:29.285009 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b7f783-3e52-4bbe-a7dd-c8620f1d27b9 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:29.307593 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06dd990-35aa-4f92-afed-64ccaed574d8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:29.309021 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06dd990-35aa-4f92-afed-64ccaed574d8 service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:29.310246 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06dd990-35aa-4f92-afed-64ccaed574d8 service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:29.310840 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06dd990-35aa-4f92-afed-64ccaed574d8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:29.614617 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06dd990-35aa-4f92-afed-64ccaed574d8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:29.616974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06dd990-35aa-4f92-afed-64ccaed574d8 service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:29.640314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 882/1744] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:08:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 757 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 31 01:08:29.757422 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3887d03-a2f5-4c7b-afef-5b549de675cd service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:29.759972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c3887d03-a2f5-4c7b-afef-5b549de675cd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:29.798667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3887d03-a2f5-4c7b-afef-5b549de675cd service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:29.841275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:29.844079 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:29.844079 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:29.844079 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] PATH_INFO: `/v3/projects/4449fa3a4fcb4f9598612b3ff3a54cbc` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:29.850353 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] RBAC: Authorizing `identity:delete_project(project_id=4449fa3a4fcb4f9598612b3ff3a54cbc)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:29.854216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:30.079489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4449fa3a4fcb4f9598612b3ff3a54cbc', 'request_id': 'req-9a3ff761-2fee-465e-85de-0bede5205bba'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:30.105706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4449fa3a4fcb4f9598612b3ff3a54cbc', 'request_id': 'req-9a3ff761-2fee-465e-85de-0bede5205bba'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:30.138932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4449fa3a4fcb4f9598612b3ff3a54cbc', 'request_id': 'req-9a3ff761-2fee-465e-85de-0bede5205bba'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:30.165981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] The token cache is being invalidate because project 4449fa3a4fcb4f9598612b3ff3a54cbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:30.166272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:30.167160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:30.187939 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 863/1745] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:29 2022] DELETE /identity/v3/projects/4449fa3a4fcb4f9598612b3ff3a54cbc => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:30.194477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06dd990-35aa-4f92-afed-64ccaed574d8 service swift] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:30.214431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06dd990-35aa-4f92-afed-64ccaed574d8 service swift] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:30.340936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef49cec3-c084-4fca-b1b2-d23be1b335c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:30.341689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef49cec3-c084-4fca-b1b2-d23be1b335c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:30.341901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef49cec3-c084-4fca-b1b2-d23be1b335c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:30.342183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef49cec3-c084-4fca-b1b2-d23be1b335c2 admin admin] PATH_INFO: `/v3/users/1bf4dea1e8d349e4b19cc83b99e1423e` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:30.343629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef49cec3-c084-4fca-b1b2-d23be1b335c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1bf4dea1e8d349e4b19cc83b99e1423e)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:30.345650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef49cec3-c084-4fca-b1b2-d23be1b335c2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:30.371386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:30.413593 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ef49cec3-c084-4fca-b1b2-d23be1b335c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1bf4dea1e8d349e4b19cc83b99e1423e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:30.414306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ef49cec3-c084-4fca-b1b2-d23be1b335c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1bf4dea1e8d349e4b19cc83b99e1423e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:30.426813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ef49cec3-c084-4fca-b1b2-d23be1b335c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1bf4dea1e8d349e4b19cc83b99e1423e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:30.443450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ef49cec3-c084-4fca-b1b2-d23be1b335c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1bf4dea1e8d349e4b19cc83b99e1423e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:30.450995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ef49cec3-c084-4fca-b1b2-d23be1b335c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1bf4dea1e8d349e4b19cc83b99e1423e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:30.458330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 883/1746] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:30 2022] DELETE /identity/v3/users/1bf4dea1e8d349e4b19cc83b99e1423e => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:31.003551 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:31.005325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:31.027781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3ff761-2fee-465e-85de-0bede5205bba admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:31.096969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:31.105142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca382c7-77d9-4a02-b206-aed92fbd2396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:31.106340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca382c7-77d9-4a02-b206-aed92fbd2396 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:31.106629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca382c7-77d9-4a02-b206-aed92fbd2396 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:31.106911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca382c7-77d9-4a02-b206-aed92fbd2396 admin admin] PATH_INFO: `/v3/projects/095b7ce8ef90456f8c9de94e187f94f8` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:31.108620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca382c7-77d9-4a02-b206-aed92fbd2396 admin admin] RBAC: Authorizing `identity:delete_project(project_id=095b7ce8ef90456f8c9de94e187f94f8)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:31.111123 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca382c7-77d9-4a02-b206-aed92fbd2396 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:31.165228 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-3ca382c7-77d9-4a02-b206-aed92fbd2396 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '095b7ce8ef90456f8c9de94e187f94f8', 'request_id': 'req-3ca382c7-77d9-4a02-b206-aed92fbd2396'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:31.171899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-3ca382c7-77d9-4a02-b206-aed92fbd2396 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '095b7ce8ef90456f8c9de94e187f94f8', 'request_id': 'req-3ca382c7-77d9-4a02-b206-aed92fbd2396'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:31.178141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-3ca382c7-77d9-4a02-b206-aed92fbd2396 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '095b7ce8ef90456f8c9de94e187f94f8', 'request_id': 'req-3ca382c7-77d9-4a02-b206-aed92fbd2396'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:31.192968 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-3ca382c7-77d9-4a02-b206-aed92fbd2396 admin admin] The token cache is being invalidate because project 095b7ce8ef90456f8c9de94e187f94f8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:31.193254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-3ca382c7-77d9-4a02-b206-aed92fbd2396 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:31.194054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-3ca382c7-77d9-4a02-b206-aed92fbd2396 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:31.195763 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 864/1747] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:31 2022] DELETE /identity/v3/projects/095b7ce8ef90456f8c9de94e187f94f8 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:31.200914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c9a9d5-8b57-4034-9673-ca92cb370eba None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:31.201202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c9a9d5-8b57-4034-9673-ca92cb370eba None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:31.201511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c9a9d5-8b57-4034-9673-ca92cb370eba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:31.223936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-04c9a9d5-8b57-4034-9673-ca92cb370eba None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:31.225186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-04c9a9d5-8b57-4034-9673-ca92cb370eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:31.294974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 884/1748] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:31.308712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37106260-4fd1-4b56-a292-7dea71a210d8 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:31.309455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37106260-4fd1-4b56-a292-7dea71a210d8 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:31.309932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37106260-4fd1-4b56-a292-7dea71a210d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:31.342370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-37106260-4fd1-4b56-a292-7dea71a210d8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:31.344077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-37106260-4fd1-4b56-a292-7dea71a210d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:31.355136 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 865/1749] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:31.360836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c9a9d5-8b57-4034-9673-ca92cb370eba None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:31.373369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c9a9d5-8b57-4034-9673-ca92cb370eba None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:31.379202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea31015e-5cd6-4225-8760-78de9ca726b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:31.379880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea31015e-5cd6-4225-8760-78de9ca726b1 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:31.380390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea31015e-5cd6-4225-8760-78de9ca726b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:31.380708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea31015e-5cd6-4225-8760-78de9ca726b1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:31.381386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea31015e-5cd6-4225-8760-78de9ca726b1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:08:31.381691 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea31015e-5cd6-4225-8760-78de9ca726b1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:31.383394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea31015e-5cd6-4225-8760-78de9ca726b1 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:31.407501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 885/1750] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:08:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:08:31.409551 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37106260-4fd1-4b56-a292-7dea71a210d8 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:31.422397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37106260-4fd1-4b56-a292-7dea71a210d8 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:31.429755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ecc53c-9cde-4ef0-ae9f-2d3b601e5eff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:31.430586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ecc53c-9cde-4ef0-ae9f-2d3b601e5eff admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:31.430870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ecc53c-9cde-4ef0-ae9f-2d3b601e5eff admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:31.431151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ecc53c-9cde-4ef0-ae9f-2d3b601e5eff admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:31.431896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ecc53c-9cde-4ef0-ae9f-2d3b601e5eff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:31.433882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ecc53c-9cde-4ef0-ae9f-2d3b601e5eff admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:31.487800 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 866/1751] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:31 2022] POST /identity/v3/projects => generated 373 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:31.489839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea31015e-5cd6-4225-8760-78de9ca726b1 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:31.501574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea31015e-5cd6-4225-8760-78de9ca726b1 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:31.579772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c756e3e6-86b3-4462-aa6d-59ce5e7ccd24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:31.581113 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c756e3e6-86b3-4462-aa6d-59ce5e7ccd24 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:31.581491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c756e3e6-86b3-4462-aa6d-59ce5e7ccd24 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:31.581849 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c756e3e6-86b3-4462-aa6d-59ce5e7ccd24 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:31.582588 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c756e3e6-86b3-4462-aa6d-59ce5e7ccd24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:31.584697 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c756e3e6-86b3-4462-aa6d-59ce5e7ccd24 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:31.608535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 886/1752] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:31 2022] POST /identity/v3/users => generated 287 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:31.613565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ecc53c-9cde-4ef0-ae9f-2d3b601e5eff admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:31.625184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ecc53c-9cde-4ef0-ae9f-2d3b601e5eff admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:31.630597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b411d2f9-3f0c-410b-9777-6c5b6629493a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:31.631237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b411d2f9-3f0c-410b-9777-6c5b6629493a admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:31.631584 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b411d2f9-3f0c-410b-9777-6c5b6629493a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:31.631860 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b411d2f9-3f0c-410b-9777-6c5b6629493a admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:31.632571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b411d2f9-3f0c-410b-9777-6c5b6629493a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:31.634233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b411d2f9-3f0c-410b-9777-6c5b6629493a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:31.646387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 867/1753] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:31 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:31.651181 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c756e3e6-86b3-4462-aa6d-59ce5e7ccd24 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:31.662016 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c756e3e6-86b3-4462-aa6d-59ce5e7ccd24 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:31.667098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90b4344-2920-400e-ae49-f41881ceaaf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:31.667763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90b4344-2920-400e-ae49-f41881ceaaf8 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:31.668048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90b4344-2920-400e-ae49-f41881ceaaf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:31.668324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90b4344-2920-400e-ae49-f41881ceaaf8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:31.669036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90b4344-2920-400e-ae49-f41881ceaaf8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:31.670626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90b4344-2920-400e-ae49-f41881ceaaf8 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:31.676654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:31.683637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 887/1754] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:31 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:31.688896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b411d2f9-3f0c-410b-9777-6c5b6629493a admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:31.701058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b411d2f9-3f0c-410b-9777-6c5b6629493a admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:31.706901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd330cd4-b356-41bb-aaff-65e30b1cedaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:31.707751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd330cd4-b356-41bb-aaff-65e30b1cedaa admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:31.708181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd330cd4-b356-41bb-aaff-65e30b1cedaa admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:31.708518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd330cd4-b356-41bb-aaff-65e30b1cedaa admin admin] PATH_INFO: `/v3/projects/bd337d5066d649d7a1233e454bf73c3e/users/bc8c26db74024f049674334e3519d3f5/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:31.728539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd330cd4-b356-41bb-aaff-65e30b1cedaa admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd337d5066d649d7a1233e454bf73c3e, user_id=bc8c26db74024f049674334e3519d3f5, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:31.731553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd330cd4-b356-41bb-aaff-65e30b1cedaa admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:31.751328 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 868/1755] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:31 2022] PUT /identity/v3/projects/bd337d5066d649d7a1233e454bf73c3e/users/bc8c26db74024f049674334e3519d3f5/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:31.758433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e4a3a4-5ad3-48e5-80e4-f0a0e52e3a7a None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:31.758917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e4a3a4-5ad3-48e5-80e4-f0a0e52e3a7a None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:31.759446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e4a3a4-5ad3-48e5-80e4-f0a0e52e3a7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:31.786265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-52e4a3a4-5ad3-48e5-80e4-f0a0e52e3a7a None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:31.788636 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-52e4a3a4-5ad3-48e5-80e4-f0a0e52e3a7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:31.873956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 888/1756] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:31.894502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd330cd4-b356-41bb-aaff-65e30b1cedaa admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:31.897244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fd330cd4-b356-41bb-aaff-65e30b1cedaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:32.030455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd330cd4-b356-41bb-aaff-65e30b1cedaa admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:32.037395 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd65645-4f6b-4461-b881-1e5c9099a715 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:32.038459 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd65645-4f6b-4461-b881-1e5c9099a715 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:32.038943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd65645-4f6b-4461-b881-1e5c9099a715 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:32.039465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd65645-4f6b-4461-b881-1e5c9099a715 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:32.069455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd65645-4f6b-4461-b881-1e5c9099a715 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:32.072114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd65645-4f6b-4461-b881-1e5c9099a715 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:32.098084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 869/1757] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:33.437562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4aa1fd-70cd-4caa-9128-b092248ee445 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:33.438844 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4aa1fd-70cd-4caa-9128-b092248ee445 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:33.440181 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4aa1fd-70cd-4caa-9128-b092248ee445 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:33.501103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ac4aa1fd-70cd-4caa-9128-b092248ee445 None None] MFA Rules not processed for user `bc8c26db74024f049674334e3519d3f5`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:33.506088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ac4aa1fd-70cd-4caa-9128-b092248ee445 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:33.640119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 889/1758] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:33 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:33.641379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd65645-4f6b-4461-b881-1e5c9099a715 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:33.670453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd65645-4f6b-4461-b881-1e5c9099a715 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:33.697377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efb0843-a9ee-4761-9ed6-d00229f9f9cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:33.701294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efb0843-a9ee-4761-9ed6-d00229f9f9cf admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:33.701730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efb0843-a9ee-4761-9ed6-d00229f9f9cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:33.702204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efb0843-a9ee-4761-9ed6-d00229f9f9cf admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:33.703653 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efb0843-a9ee-4761-9ed6-d00229f9f9cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:33.708615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efb0843-a9ee-4761-9ed6-d00229f9f9cf admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:33.755798 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 870/1759] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:33 2022] POST /identity/v3/projects => generated 373 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:33.760183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4aa1fd-70cd-4caa-9128-b092248ee445 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:33.782437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4aa1fd-70cd-4caa-9128-b092248ee445 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:33.892058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5ae5dc-c8ed-42c8-93a1-5f603b7712ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:33.893464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5ae5dc-c8ed-42c8-93a1-5f603b7712ea admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:33.893803 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5ae5dc-c8ed-42c8-93a1-5f603b7712ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:33.894139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5ae5dc-c8ed-42c8-93a1-5f603b7712ea admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:33.894899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5ae5dc-c8ed-42c8-93a1-5f603b7712ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:33.897636 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5ae5dc-c8ed-42c8-93a1-5f603b7712ea admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:33.946414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 890/1760] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:33 2022] POST /identity/v3/users => generated 293 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:33.952345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efb0843-a9ee-4761-9ed6-d00229f9f9cf admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:33.966098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:34.003422 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efb0843-a9ee-4761-9ed6-d00229f9f9cf admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:34.006235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba4dab5-de70-4c02-a7a1-f7881ec1d9f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:34.007702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba4dab5-de70-4c02-a7a1-f7881ec1d9f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:34.008236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba4dab5-de70-4c02-a7a1-f7881ec1d9f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:34.008668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba4dab5-de70-4c02-a7a1-f7881ec1d9f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:34.009625 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba4dab5-de70-4c02-a7a1-f7881ec1d9f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:34.011881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba4dab5-de70-4c02-a7a1-f7881ec1d9f0 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:34.043805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 871/1761] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:33 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:34.049925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5ae5dc-c8ed-42c8-93a1-5f603b7712ea admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:34.071736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5ae5dc-c8ed-42c8-93a1-5f603b7712ea admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:34.089092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51835b75-c5c7-469e-ad97-1c6a3bc6f2d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:34.089858 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51835b75-c5c7-469e-ad97-1c6a3bc6f2d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:34.094395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51835b75-c5c7-469e-ad97-1c6a3bc6f2d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:34.095511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51835b75-c5c7-469e-ad97-1c6a3bc6f2d2 admin admin] PATH_INFO: `/v3/projects/5baf560f78b446f691b16d77fdc05407/users/376158b98a414cf78dceb70ba4f357c9/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:34.147924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51835b75-c5c7-469e-ad97-1c6a3bc6f2d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5baf560f78b446f691b16d77fdc05407, user_id=376158b98a414cf78dceb70ba4f357c9, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:34.151569 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51835b75-c5c7-469e-ad97-1c6a3bc6f2d2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:34.156001 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:34.197732 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 891/1762] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:34 2022] PUT /identity/v3/projects/5baf560f78b446f691b16d77fdc05407/users/376158b98a414cf78dceb70ba4f357c9/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:34.209157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba4dab5-de70-4c02-a7a1-f7881ec1d9f0 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:34.229469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba4dab5-de70-4c02-a7a1-f7881ec1d9f0 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:34.392024 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ff6180-fd25-4bbe-bff4-e51d77797cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:34.394178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ff6180-fd25-4bbe-bff4-e51d77797cf8 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:34.394540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ff6180-fd25-4bbe-bff4-e51d77797cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:34.394748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ff6180-fd25-4bbe-bff4-e51d77797cf8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:34.395685 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ff6180-fd25-4bbe-bff4-e51d77797cf8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:34.398014 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ff6180-fd25-4bbe-bff4-e51d77797cf8 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:34.411994 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 872/1763] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:34 2022] GET /identity/v3/roles => generated 1420 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:34.418723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51835b75-c5c7-469e-ad97-1c6a3bc6f2d2 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:34.441361 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51835b75-c5c7-469e-ad97-1c6a3bc6f2d2 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:34.450044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d79d99-f607-4398-8d46-4e9e27786f1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:34.451511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d79d99-f607-4398-8d46-4e9e27786f1c admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:34.452038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d79d99-f607-4398-8d46-4e9e27786f1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:34.452587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d79d99-f607-4398-8d46-4e9e27786f1c admin admin] PATH_INFO: `/v3/projects/5baf560f78b446f691b16d77fdc05407/users/376158b98a414cf78dceb70ba4f357c9/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:34.456572 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d79d99-f607-4398-8d46-4e9e27786f1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=5baf560f78b446f691b16d77fdc05407, user_id=376158b98a414cf78dceb70ba4f357c9, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:34.462726 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d79d99-f607-4398-8d46-4e9e27786f1c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:34.487909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 892/1764] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:34 2022] PUT /identity/v3/projects/5baf560f78b446f691b16d77fdc05407/users/376158b98a414cf78dceb70ba4f357c9/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:35.583390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887e07f9-d687-46b2-93ad-f6c0681d5725 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:35.584184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887e07f9-d687-46b2-93ad-f6c0681d5725 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:35.584184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887e07f9-d687-46b2-93ad-f6c0681d5725 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:35.618784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-887e07f9-d687-46b2-93ad-f6c0681d5725 None None] MFA Rules not processed for user `376158b98a414cf78dceb70ba4f357c9`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:35.620082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-887e07f9-d687-46b2-93ad-f6c0681d5725 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:35.761029 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 873/1765] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:35 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:35.780727 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d79d99-f607-4398-8d46-4e9e27786f1c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:35.782871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-65d79d99-f607-4398-8d46-4e9e27786f1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:35.860029 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d79d99-f607-4398-8d46-4e9e27786f1c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:35.863831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f05d635-8277-49f8-a8d4-91001a10850b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:35.864454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05d635-8277-49f8-a8d4-91001a10850b service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:35.864723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05d635-8277-49f8-a8d4-91001a10850b service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:35.864995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05d635-8277-49f8-a8d4-91001a10850b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:35.937203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f05d635-8277-49f8-a8d4-91001a10850b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:35.940646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f05d635-8277-49f8-a8d4-91001a10850b service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:35.960027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 893/1766] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:35 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:08:42.154914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887e07f9-d687-46b2-93ad-f6c0681d5725 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:42.156503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-887e07f9-d687-46b2-93ad-f6c0681d5725 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:42.173543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887e07f9-d687-46b2-93ad-f6c0681d5725 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:42.180109 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f95ccdd-dd59-4be4-8a63-10f687c8bbcc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:42.181508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f95ccdd-dd59-4be4-8a63-10f687c8bbcc service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:42.182295 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f95ccdd-dd59-4be4-8a63-10f687c8bbcc service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:42.182970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f95ccdd-dd59-4be4-8a63-10f687c8bbcc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:42.208271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f95ccdd-dd59-4be4-8a63-10f687c8bbcc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:42.209834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f95ccdd-dd59-4be4-8a63-10f687c8bbcc service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:42.237992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 874/1767] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:42 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:08:42.846221 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f05d635-8277-49f8-a8d4-91001a10850b service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:42.847886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3f05d635-8277-49f8-a8d4-91001a10850b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:43.149912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f05d635-8277-49f8-a8d4-91001a10850b service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:43.177129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865ac99d-d1c7-4ad3-982d-ff51159b01a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:43.178256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865ac99d-d1c7-4ad3-982d-ff51159b01a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:43.178728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865ac99d-d1c7-4ad3-982d-ff51159b01a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:43.179321 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865ac99d-d1c7-4ad3-982d-ff51159b01a2 admin admin] PATH_INFO: `/v3/users/dfb41ea5723c416ea0969e214625b94b` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:43.202977 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865ac99d-d1c7-4ad3-982d-ff51159b01a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dfb41ea5723c416ea0969e214625b94b)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:43.205773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865ac99d-d1c7-4ad3-982d-ff51159b01a2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:43.243582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8841e9-8d33-4d9d-a278-322cfef4a2f1 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:43.247846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8841e9-8d33-4d9d-a278-322cfef4a2f1 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:43.247846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8841e9-8d33-4d9d-a278-322cfef4a2f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:43.325749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-5c8841e9-8d33-4d9d-a278-322cfef4a2f1 None None] MFA Rules not processed for user `94e868131d33472b9ded8486274a07b9`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:43.329572 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5c8841e9-8d33-4d9d-a278-322cfef4a2f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:43.371918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:43.413910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-865ac99d-d1c7-4ad3-982d-ff51159b01a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dfb41ea5723c416ea0969e214625b94b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:43.417030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-865ac99d-d1c7-4ad3-982d-ff51159b01a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dfb41ea5723c416ea0969e214625b94b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:43.453212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-865ac99d-d1c7-4ad3-982d-ff51159b01a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dfb41ea5723c416ea0969e214625b94b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:43.544440 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-865ac99d-d1c7-4ad3-982d-ff51159b01a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dfb41ea5723c416ea0969e214625b94b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:43.590562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-865ac99d-d1c7-4ad3-982d-ff51159b01a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dfb41ea5723c416ea0969e214625b94b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:43.608582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 894/1768] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:42 2022] DELETE /identity/v3/users/dfb41ea5723c416ea0969e214625b94b => generated 0 bytes in 763 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:43.620151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7935f184-2f40-4ab2-adbd-25db07f2bc60 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:43.622024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7935f184-2f40-4ab2-adbd-25db07f2bc60 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:43.623942 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7935f184-2f40-4ab2-adbd-25db07f2bc60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:43.637627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 875/1769] 10.222.0.64 () {58 vars in 1015 bytes} [Sun Jul 31 01:08:43 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 396 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:43.651159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8841e9-8d33-4d9d-a278-322cfef4a2f1 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:43.663911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5c8841e9-8d33-4d9d-a278-322cfef4a2f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:43.726119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8841e9-8d33-4d9d-a278-322cfef4a2f1 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:43.744051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-7935f184-2f40-4ab2-adbd-25db07f2bc60 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:43.745505 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7935f184-2f40-4ab2-adbd-25db07f2bc60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:43.766143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:43.941771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 895/1770] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 327 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:43.943734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7935f184-2f40-4ab2-adbd-25db07f2bc60 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:43.946157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7935f184-2f40-4ab2-adbd-25db07f2bc60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:43.965112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7935f184-2f40-4ab2-adbd-25db07f2bc60 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:43.974857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e754fe9-70a2-468c-b011-90287e398d5d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:43.976158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e754fe9-70a2-468c-b011-90287e398d5d service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:43.977309 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e754fe9-70a2-468c-b011-90287e398d5d service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:43.978160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eaa5966-26f2-4577-99a3-0ea96cfc45f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:43.978697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e754fe9-70a2-468c-b011-90287e398d5d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:43.980416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eaa5966-26f2-4577-99a3-0ea96cfc45f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:43.980899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eaa5966-26f2-4577-99a3-0ea96cfc45f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:43.982021 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eaa5966-26f2-4577-99a3-0ea96cfc45f5 admin admin] PATH_INFO: `/v3/users/2d6e967d04db459090cfdadc92e8703b` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:43.988381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eaa5966-26f2-4577-99a3-0ea96cfc45f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d6e967d04db459090cfdadc92e8703b)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:43.990972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eaa5966-26f2-4577-99a3-0ea96cfc45f5 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:44.000764 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e754fe9-70a2-468c-b011-90287e398d5d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:44.002593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e754fe9-70a2-468c-b011-90287e398d5d service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:44.030124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 876/1771] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:43 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:08:44.040794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e754fe9-70a2-468c-b011-90287e398d5d service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:44.064612 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8e754fe9-70a2-468c-b011-90287e398d5d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:44.129830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e754fe9-70a2-468c-b011-90287e398d5d service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:44.147963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3235f56a-8864-48f4-ac50-b40af28d9c13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:44.150174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3235f56a-8864-48f4-ac50-b40af28d9c13 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:44.150404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7eaa5966-26f2-4577-99a3-0ea96cfc45f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d6e967d04db459090cfdadc92e8703b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:44.154576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3235f56a-8864-48f4-ac50-b40af28d9c13 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:44.155031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7eaa5966-26f2-4577-99a3-0ea96cfc45f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d6e967d04db459090cfdadc92e8703b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:44.155412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3235f56a-8864-48f4-ac50-b40af28d9c13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:44.172279 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7eaa5966-26f2-4577-99a3-0ea96cfc45f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d6e967d04db459090cfdadc92e8703b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:44.220883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7eaa5966-26f2-4577-99a3-0ea96cfc45f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d6e967d04db459090cfdadc92e8703b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:44.224950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3235f56a-8864-48f4-ac50-b40af28d9c13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:44.228202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3235f56a-8864-48f4-ac50-b40af28d9c13 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:44.246920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7eaa5966-26f2-4577-99a3-0ea96cfc45f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d6e967d04db459090cfdadc92e8703b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:44.263329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 896/1772] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:43 2022] DELETE /identity/v3/users/2d6e967d04db459090cfdadc92e8703b => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:44.270503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621f5a9a-5eb8-4bc6-945d-bdb59f8f1c46 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:44.271112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621f5a9a-5eb8-4bc6-945d-bdb59f8f1c46 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:44.271605 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621f5a9a-5eb8-4bc6-945d-bdb59f8f1c46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:44.299679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 877/1773] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:44.325451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-621f5a9a-5eb8-4bc6-945d-bdb59f8f1c46 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:44.327492 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-621f5a9a-5eb8-4bc6-945d-bdb59f8f1c46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:44.388978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:44.491639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 897/1774] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:44.515474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3235f56a-8864-48f4-ac50-b40af28d9c13 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:44.559101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3235f56a-8864-48f4-ac50-b40af28d9c13 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:44.731772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c7df91-6404-4f23-bb85-228e4304cee5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:44.732570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c7df91-6404-4f23-bb85-228e4304cee5 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:44.732955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c7df91-6404-4f23-bb85-228e4304cee5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:44.733333 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c7df91-6404-4f23-bb85-228e4304cee5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:44.777813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c7df91-6404-4f23-bb85-228e4304cee5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:44.779728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c7df91-6404-4f23-bb85-228e4304cee5 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:44.793284 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 878/1775] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:44.993114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621f5a9a-5eb8-4bc6-945d-bdb59f8f1c46 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:44.995633 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-621f5a9a-5eb8-4bc6-945d-bdb59f8f1c46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:45.018365 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621f5a9a-5eb8-4bc6-945d-bdb59f8f1c46 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:45.029205 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a38d768-b615-4555-a085-438e1d508f52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:45.030676 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38d768-b615-4555-a085-438e1d508f52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:45.031213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38d768-b615-4555-a085-438e1d508f52 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:45.031763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38d768-b615-4555-a085-438e1d508f52 admin admin] PATH_INFO: `/v3/projects/470f049fa00841c98ce7422f3c5e423e` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:45.034697 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a38d768-b615-4555-a085-438e1d508f52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=470f049fa00841c98ce7422f3c5e423e)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:45.038771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a38d768-b615-4555-a085-438e1d508f52 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:45.193770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0a38d768-b615-4555-a085-438e1d508f52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '470f049fa00841c98ce7422f3c5e423e', 'request_id': 'req-0a38d768-b615-4555-a085-438e1d508f52'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.203973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0a38d768-b615-4555-a085-438e1d508f52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '470f049fa00841c98ce7422f3c5e423e', 'request_id': 'req-0a38d768-b615-4555-a085-438e1d508f52'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.213539 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0a38d768-b615-4555-a085-438e1d508f52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '470f049fa00841c98ce7422f3c5e423e', 'request_id': 'req-0a38d768-b615-4555-a085-438e1d508f52'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.234442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0a38d768-b615-4555-a085-438e1d508f52 admin admin] The token cache is being invalidate because project 470f049fa00841c98ce7422f3c5e423e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:45.234980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0a38d768-b615-4555-a085-438e1d508f52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.236840 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0a38d768-b615-4555-a085-438e1d508f52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.239931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 898/1776] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:44 2022] DELETE /identity/v3/projects/470f049fa00841c98ce7422f3c5e423e => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:45.244264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c7df91-6404-4f23-bb85-228e4304cee5 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:45.260917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c7df91-6404-4f23-bb85-228e4304cee5 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:45.360396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c07afab-552f-407b-8183-c10f4542f01c None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:45.360681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c07afab-552f-407b-8183-c10f4542f01c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:45.361143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c07afab-552f-407b-8183-c10f4542f01c None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:45.362921 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 899/1777] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:08:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:08:45.402021 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f79869f-f7b8-463c-a979-3d694ddecab8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:45.402773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f79869f-f7b8-463c-a979-3d694ddecab8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:45.403064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f79869f-f7b8-463c-a979-3d694ddecab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:45.403918 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:45.405245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:45.405816 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:45.406353 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] PATH_INFO: `/v3/users/e9cff3b66be64f4cb3ad7e88a6e88267` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:45.410664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9cff3b66be64f4cb3ad7e88a6e88267)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:45.414796 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:45.430219 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-5f79869f-f7b8-463c-a979-3d694ddecab8 None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:45.431526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5f79869f-f7b8-463c-a979-3d694ddecab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:45.555320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:45.563095 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9cff3b66be64f4cb3ad7e88a6e88267'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.579144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9cff3b66be64f4cb3ad7e88a6e88267'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.580691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9cff3b66be64f4cb3ad7e88a6e88267'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.589186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 900/1778] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:08:45 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:45.591794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f79869f-f7b8-463c-a979-3d694ddecab8 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:45.594323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5f79869f-f7b8-463c-a979-3d694ddecab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:45.626405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9cff3b66be64f4cb3ad7e88a6e88267'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.626405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9cff3b66be64f4cb3ad7e88a6e88267'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.627612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f79869f-f7b8-463c-a979-3d694ddecab8 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:45.638279 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ccaf32-fdd5-4369-9b2a-a44976ed0482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:45.639695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ccaf32-fdd5-4369-9b2a-a44976ed0482 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:45.640294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ccaf32-fdd5-4369-9b2a-a44976ed0482 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:45.640957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ccaf32-fdd5-4369-9b2a-a44976ed0482 admin admin] PATH_INFO: `/v3/projects/4c82effc151742818916d6ff0431832b` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:45.644542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ccaf32-fdd5-4369-9b2a-a44976ed0482 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c82effc151742818916d6ff0431832b)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:45.648522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ccaf32-fdd5-4369-9b2a-a44976ed0482 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:45.657038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 879/1779] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:45 2022] DELETE /identity/v3/users/e9cff3b66be64f4cb3ad7e88a6e88267 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:45.657038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:45.657038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:45.739299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:45.770730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-02ccaf32-fdd5-4369-9b2a-a44976ed0482 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c82effc151742818916d6ff0431832b', 'request_id': 'req-02ccaf32-fdd5-4369-9b2a-a44976ed0482'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.786443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-02ccaf32-fdd5-4369-9b2a-a44976ed0482 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c82effc151742818916d6ff0431832b', 'request_id': 'req-02ccaf32-fdd5-4369-9b2a-a44976ed0482'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.803435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-02ccaf32-fdd5-4369-9b2a-a44976ed0482 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c82effc151742818916d6ff0431832b', 'request_id': 'req-02ccaf32-fdd5-4369-9b2a-a44976ed0482'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.838611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-02ccaf32-fdd5-4369-9b2a-a44976ed0482 admin admin] The token cache is being invalidate because project 4c82effc151742818916d6ff0431832b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:45.839265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-02ccaf32-fdd5-4369-9b2a-a44976ed0482 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.840550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-02ccaf32-fdd5-4369-9b2a-a44976ed0482 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:45.843720 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 901/1780] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:45 2022] DELETE /identity/v3/projects/4c82effc151742818916d6ff0431832b => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:45.851854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435a3970-a1c8-400e-80e3-5ac7d4f28725 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:45.852447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435a3970-a1c8-400e-80e3-5ac7d4f28725 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:45.853346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435a3970-a1c8-400e-80e3-5ac7d4f28725 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:45.884374 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4887980d-b7ce-441e-9332-ae21cd6a1123 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:45.891051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-435a3970-a1c8-400e-80e3-5ac7d4f28725 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:45.894304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-435a3970-a1c8-400e-80e3-5ac7d4f28725 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:45.984845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc1dd8-46cc-4528-a6f2-a69926581f8f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:45.985508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc1dd8-46cc-4528-a6f2-a69926581f8f service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:45.985828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc1dd8-46cc-4528-a6f2-a69926581f8f service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:45.986129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc1dd8-46cc-4528-a6f2-a69926581f8f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:45.988636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9bfc1dd8-46cc-4528-a6f2-a69926581f8f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:46.004797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfc1dd8-46cc-4528-a6f2-a69926581f8f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.006500 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfc1dd8-46cc-4528-a6f2-a69926581f8f service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.020660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 880/1781] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:45 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:46.032366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 902/1782] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:46.046081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14aedc0-db8b-4777-a63a-0905b00accd4 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.046763 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14aedc0-db8b-4777-a63a-0905b00accd4 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.047265 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14aedc0-db8b-4777-a63a-0905b00accd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.083471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d14aedc0-db8b-4777-a63a-0905b00accd4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:46.084721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d14aedc0-db8b-4777-a63a-0905b00accd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:46.095435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 881/1783] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:46.101862 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435a3970-a1c8-400e-80e3-5ac7d4f28725 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.131503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435a3970-a1c8-400e-80e3-5ac7d4f28725 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:46.140843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438b5b9a-ffe2-4973-87d6-e23d8485c745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:46.141972 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438b5b9a-ffe2-4973-87d6-e23d8485c745 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.142481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438b5b9a-ffe2-4973-87d6-e23d8485c745 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.142987 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438b5b9a-ffe2-4973-87d6-e23d8485c745 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.145123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438b5b9a-ffe2-4973-87d6-e23d8485c745 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:08:46.151583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438b5b9a-ffe2-4973-87d6-e23d8485c745 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.153584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438b5b9a-ffe2-4973-87d6-e23d8485c745 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.179401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 903/1784] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:08:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:08:46.185127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14aedc0-db8b-4777-a63a-0905b00accd4 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.201742 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14aedc0-db8b-4777-a63a-0905b00accd4 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:46.210002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3aa99ee-6c94-4603-900d-9d08681a2e33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:46.211727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aa99ee-6c94-4603-900d-9d08681a2e33 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.212201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aa99ee-6c94-4603-900d-9d08681a2e33 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.212692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aa99ee-6c94-4603-900d-9d08681a2e33 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.213941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3aa99ee-6c94-4603-900d-9d08681a2e33 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.217817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3aa99ee-6c94-4603-900d-9d08681a2e33 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.218536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438b5b9a-ffe2-4973-87d6-e23d8485c745 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.221300 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-438b5b9a-ffe2-4973-87d6-e23d8485c745 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:46.245612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438b5b9a-ffe2-4973-87d6-e23d8485c745 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:46.253909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506efd23-f373-498c-a165-b8ee618437c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:46.254859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506efd23-f373-498c-a165-b8ee618437c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.255250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506efd23-f373-498c-a165-b8ee618437c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.255666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506efd23-f373-498c-a165-b8ee618437c0 admin admin] PATH_INFO: `/v3/projects/7ad4673ebcbc464b802a95670bb03f45` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.257361 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506efd23-f373-498c-a165-b8ee618437c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7ad4673ebcbc464b802a95670bb03f45)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.259532 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506efd23-f373-498c-a165-b8ee618437c0 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.283955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 882/1785] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:46 2022] POST /identity/v3/projects => generated 371 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:46.290144 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3aa99ee-6c94-4603-900d-9d08681a2e33 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.327674 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:DELETE=4,keystone:SELECT=94,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:46.354560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-506efd23-f373-498c-a165-b8ee618437c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7ad4673ebcbc464b802a95670bb03f45', 'request_id': 'req-506efd23-f373-498c-a165-b8ee618437c0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:46.366161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-506efd23-f373-498c-a165-b8ee618437c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7ad4673ebcbc464b802a95670bb03f45', 'request_id': 'req-506efd23-f373-498c-a165-b8ee618437c0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:46.378454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-506efd23-f373-498c-a165-b8ee618437c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7ad4673ebcbc464b802a95670bb03f45', 'request_id': 'req-506efd23-f373-498c-a165-b8ee618437c0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:46.393061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3aa99ee-6c94-4603-900d-9d08681a2e33 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:46.395691 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-506efd23-f373-498c-a165-b8ee618437c0 admin admin] The token cache is being invalidate because project 7ad4673ebcbc464b802a95670bb03f45 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:46.396208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-506efd23-f373-498c-a165-b8ee618437c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:46.397627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-506efd23-f373-498c-a165-b8ee618437c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:46.400846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 904/1786] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:46 2022] DELETE /identity/v3/projects/7ad4673ebcbc464b802a95670bb03f45 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:46.409757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64cf299-ea4d-4a98-88e2-d6410a9ace27 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.411155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64cf299-ea4d-4a98-88e2-d6410a9ace27 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.412527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64cf299-ea4d-4a98-88e2-d6410a9ace27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.440335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-e64cf299-ea4d-4a98-88e2-d6410a9ace27 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:46.441528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e64cf299-ea4d-4a98-88e2-d6410a9ace27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:46.480316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:46.491910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c42f2e-fb55-4ffc-b00e-0de7b448ba7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:46.492550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c42f2e-fb55-4ffc-b00e-0de7b448ba7f admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.493077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c42f2e-fb55-4ffc-b00e-0de7b448ba7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.493358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c42f2e-fb55-4ffc-b00e-0de7b448ba7f admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.494029 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c42f2e-fb55-4ffc-b00e-0de7b448ba7f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.496083 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c42f2e-fb55-4ffc-b00e-0de7b448ba7f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.525969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 883/1787] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:46 2022] POST /identity/v3/users => generated 286 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:46.526160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c42f2e-fb55-4ffc-b00e-0de7b448ba7f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.527906 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f6c42f2e-fb55-4ffc-b00e-0de7b448ba7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:46.541113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c42f2e-fb55-4ffc-b00e-0de7b448ba7f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:46.543646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 905/1788] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:46.546858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1460df3a-9fcf-477d-a537-3bf1ea14d817 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:46.547967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1460df3a-9fcf-477d-a537-3bf1ea14d817 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.547967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1460df3a-9fcf-477d-a537-3bf1ea14d817 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.548323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1460df3a-9fcf-477d-a537-3bf1ea14d817 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.549368 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1460df3a-9fcf-477d-a537-3bf1ea14d817 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.551075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1460df3a-9fcf-477d-a537-3bf1ea14d817 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.556232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffc0026-902d-4388-8e2e-d336f0630ca1 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.556520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffc0026-902d-4388-8e2e-d336f0630ca1 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.556785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffc0026-902d-4388-8e2e-d336f0630ca1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.568537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 884/1789] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:46 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:08:46.572672 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1460df3a-9fcf-477d-a537-3bf1ea14d817 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.579215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-9ffc0026-902d-4388-8e2e-d336f0630ca1 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:46.580615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9ffc0026-902d-4388-8e2e-d336f0630ca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:46.584764 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1460df3a-9fcf-477d-a537-3bf1ea14d817 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:46.589675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 906/1790] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:46.590041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634fab95-008c-43bb-98a1-f0687f5b07ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:46.590466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634fab95-008c-43bb-98a1-f0687f5b07ed admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.590752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634fab95-008c-43bb-98a1-f0687f5b07ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.591026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634fab95-008c-43bb-98a1-f0687f5b07ed admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.591761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634fab95-008c-43bb-98a1-f0687f5b07ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.593424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634fab95-008c-43bb-98a1-f0687f5b07ed admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.594286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffc0026-902d-4388-8e2e-d336f0630ca1 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.605058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffc0026-902d-4388-8e2e-d336f0630ca1 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:46.607080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 885/1791] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:46 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:08:46.610320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5c7e6e-dc53-4f36-83a5-ce484728f8e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:46.611566 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5c7e6e-dc53-4f36-83a5-ce484728f8e2 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.611847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5c7e6e-dc53-4f36-83a5-ce484728f8e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.612112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5c7e6e-dc53-4f36-83a5-ce484728f8e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.612839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5c7e6e-dc53-4f36-83a5-ce484728f8e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:08:46.613167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5c7e6e-dc53-4f36-83a5-ce484728f8e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.613493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634fab95-008c-43bb-98a1-f0687f5b07ed admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.614792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5c7e6e-dc53-4f36-83a5-ce484728f8e2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.624568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634fab95-008c-43bb-98a1-f0687f5b07ed admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:46.628798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 907/1792] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:08:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:08:46.631320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080f7e45-9ca5-47b5-a5d0-d318e97d23de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:46.632397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080f7e45-9ca5-47b5-a5d0-d318e97d23de admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.632761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080f7e45-9ca5-47b5-a5d0-d318e97d23de admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.633112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080f7e45-9ca5-47b5-a5d0-d318e97d23de admin admin] PATH_INFO: `/v3/projects/abb3207f94ac48b7be28ae83b234521b/users/af9ed1804b254e899eb5d7fbe4840411/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5c7e6e-dc53-4f36-83a5-ce484728f8e2 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.644603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5c7e6e-dc53-4f36-83a5-ce484728f8e2 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:46.650389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d30be9-0c34-4063-bab6-3e28b98d9125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:46.651054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d30be9-0c34-4063-bab6-3e28b98d9125 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.651334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d30be9-0c34-4063-bab6-3e28b98d9125 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.651645 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d30be9-0c34-4063-bab6-3e28b98d9125 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.652057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080f7e45-9ca5-47b5-a5d0-d318e97d23de admin admin] RBAC: Authorizing `identity:create_grant(project_id=abb3207f94ac48b7be28ae83b234521b, user_id=af9ed1804b254e899eb5d7fbe4840411, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.652480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d30be9-0c34-4063-bab6-3e28b98d9125 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.654423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d30be9-0c34-4063-bab6-3e28b98d9125 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.655320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080f7e45-9ca5-47b5-a5d0-d318e97d23de admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.679481 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 886/1793] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:46 2022] PUT /identity/v3/projects/abb3207f94ac48b7be28ae83b234521b/users/af9ed1804b254e899eb5d7fbe4840411/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:46.702006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 908/1794] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:46 2022] POST /identity/v3/projects => generated 385 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:46.702301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8487d2-a559-4d17-9303-e77456ad0807 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.702455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8487d2-a559-4d17-9303-e77456ad0807 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.702900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8487d2-a559-4d17-9303-e77456ad0807 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.707910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d30be9-0c34-4063-bab6-3e28b98d9125 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.751336 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-1b8487d2-a559-4d17-9303-e77456ad0807 None None] MFA Rules not processed for user `af9ed1804b254e899eb5d7fbe4840411`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:46.754279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1b8487d2-a559-4d17-9303-e77456ad0807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:46.845167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d30be9-0c34-4063-bab6-3e28b98d9125 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:46.854559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb76f4f-89a1-49d6-ba0f-f39a4e33bbe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:46.855675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb76f4f-89a1-49d6-ba0f-f39a4e33bbe9 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.856173 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb76f4f-89a1-49d6-ba0f-f39a4e33bbe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.856728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb76f4f-89a1-49d6-ba0f-f39a4e33bbe9 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.857659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:46.857983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb76f4f-89a1-49d6-ba0f-f39a4e33bbe9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.864847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb76f4f-89a1-49d6-ba0f-f39a4e33bbe9 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.878881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 887/1795] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:46 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:46.888994 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8487d2-a559-4d17-9303-e77456ad0807 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.903659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8487d2-a559-4d17-9303-e77456ad0807 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:46.909963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b536b65-7dfe-4778-8102-1a1b48faef87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:46.910601 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b536b65-7dfe-4778-8102-1a1b48faef87 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.911077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b536b65-7dfe-4778-8102-1a1b48faef87 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.911223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b536b65-7dfe-4778-8102-1a1b48faef87 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.911872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b536b65-7dfe-4778-8102-1a1b48faef87 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.913847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b536b65-7dfe-4778-8102-1a1b48faef87 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.916100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 909/1796] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:46 2022] POST /identity/v3/users => generated 293 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:46.921674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb76f4f-89a1-49d6-ba0f-f39a4e33bbe9 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.949607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb76f4f-89a1-49d6-ba0f-f39a4e33bbe9 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:46.960786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508229c1-3491-49bf-a0b5-4f9eb3e4448a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:46.961907 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508229c1-3491-49bf-a0b5-4f9eb3e4448a admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:46.962451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508229c1-3491-49bf-a0b5-4f9eb3e4448a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:46.963026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508229c1-3491-49bf-a0b5-4f9eb3e4448a admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:46.964627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508229c1-3491-49bf-a0b5-4f9eb3e4448a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:46.967403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508229c1-3491-49bf-a0b5-4f9eb3e4448a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:46.967568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 888/1797] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:46 2022] POST /identity/v3/projects => generated 373 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:46.974154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b536b65-7dfe-4778-8102-1a1b48faef87 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:46.982414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 910/1798] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:46 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:46.987325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508229c1-3491-49bf-a0b5-4f9eb3e4448a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:47.004200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508229c1-3491-49bf-a0b5-4f9eb3e4448a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:47.070822 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b536b65-7dfe-4778-8102-1a1b48faef87 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:47.079203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161ea905-890a-4a09-aeb4-c3032a0071e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:47.080928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161ea905-890a-4a09-aeb4-c3032a0071e5 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.081501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161ea905-890a-4a09-aeb4-c3032a0071e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.082199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161ea905-890a-4a09-aeb4-c3032a0071e5 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.084490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161ea905-890a-4a09-aeb4-c3032a0071e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:47.087384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161ea905-890a-4a09-aeb4-c3032a0071e5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:47.127947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 889/1799] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:46 2022] POST /identity/v3/users => generated 293 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:47.128256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15375b4-1a7b-4ebd-8176-16200b97716c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:47.128256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15375b4-1a7b-4ebd-8176-16200b97716c admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.128256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15375b4-1a7b-4ebd-8176-16200b97716c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.128256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15375b4-1a7b-4ebd-8176-16200b97716c admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.129210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15375b4-1a7b-4ebd-8176-16200b97716c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:47.130637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15375b4-1a7b-4ebd-8176-16200b97716c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:47.131384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161ea905-890a-4a09-aeb4-c3032a0071e5 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:47.146524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 911/1800] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:46 2022] GET /identity/v3/roles => generated 1420 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:08:47.152038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15375b4-1a7b-4ebd-8176-16200b97716c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:47.152954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161ea905-890a-4a09-aeb4-c3032a0071e5 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:47.163026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547d805a-aa1a-4479-a962-4358fe150392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:47.164213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547d805a-aa1a-4479-a962-4358fe150392 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.164905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547d805a-aa1a-4479-a962-4358fe150392 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.165643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547d805a-aa1a-4479-a962-4358fe150392 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.167007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15375b4-1a7b-4ebd-8176-16200b97716c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:47.167201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547d805a-aa1a-4479-a962-4358fe150392 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:47.170031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547d805a-aa1a-4479-a962-4358fe150392 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:47.175006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcd9121-310b-47e4-94e3-34eb914f161e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:47.175987 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcd9121-310b-47e4-94e3-34eb914f161e admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.176438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcd9121-310b-47e4-94e3-34eb914f161e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.176869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcd9121-310b-47e4-94e3-34eb914f161e admin admin] PATH_INFO: `/v3/projects/eac61647e4ee4a9699be0cd8f0be0264/users/b246c19664754525af7b2e4637257848/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.193434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 890/1801] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:47 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:08:47.199189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:47.202179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547d805a-aa1a-4479-a962-4358fe150392 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:47.209066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcd9121-310b-47e4-94e3-34eb914f161e admin admin] RBAC: Authorizing `identity:create_grant(project_id=eac61647e4ee4a9699be0cd8f0be0264, user_id=b246c19664754525af7b2e4637257848, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:47.214413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcd9121-310b-47e4-94e3-34eb914f161e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:47.222795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547d805a-aa1a-4479-a962-4358fe150392 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:47.231279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec63caa-64e1-4279-8e7f-5922a6d52d8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:47.232283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec63caa-64e1-4279-8e7f-5922a6d52d8b admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.232644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec63caa-64e1-4279-8e7f-5922a6d52d8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.233011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec63caa-64e1-4279-8e7f-5922a6d52d8b admin admin] PATH_INFO: `/v3/projects/c4d51c53bb6340f88d0b9e0614a2e51f/users/5f1fa87a36e54a9c87ec780f68da2e5d/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.247421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 912/1802] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:47 2022] PUT /identity/v3/projects/eac61647e4ee4a9699be0cd8f0be0264/users/b246c19664754525af7b2e4637257848/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:47.258007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e0bfd2-b139-4104-976b-604f41d952e2 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.258956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e0bfd2-b139-4104-976b-604f41d952e2 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.258956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e0bfd2-b139-4104-976b-604f41d952e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.264456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec63caa-64e1-4279-8e7f-5922a6d52d8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4d51c53bb6340f88d0b9e0614a2e51f, user_id=5f1fa87a36e54a9c87ec780f68da2e5d, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:47.268376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec63caa-64e1-4279-8e7f-5922a6d52d8b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:47.289361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 891/1803] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:47 2022] PUT /identity/v3/projects/c4d51c53bb6340f88d0b9e0614a2e51f/users/5f1fa87a36e54a9c87ec780f68da2e5d/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:47.294263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec63caa-64e1-4279-8e7f-5922a6d52d8b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:47.316999 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f0e0bfd2-b139-4104-976b-604f41d952e2 None None] MFA Rules not processed for user `b246c19664754525af7b2e4637257848`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:47.318689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f0e0bfd2-b139-4104-976b-604f41d952e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:47.393753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec63caa-64e1-4279-8e7f-5922a6d52d8b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:47.402076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c2da93-b12f-4a8f-9e20-df501ea5b195 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:47.402971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c2da93-b12f-4a8f-9e20-df501ea5b195 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.403480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c2da93-b12f-4a8f-9e20-df501ea5b195 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.403952 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c2da93-b12f-4a8f-9e20-df501ea5b195 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.405025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c2da93-b12f-4a8f-9e20-df501ea5b195 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:47.408447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c2da93-b12f-4a8f-9e20-df501ea5b195 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:47.426505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 892/1804] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:47 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:08:47.428442 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c2da93-b12f-4a8f-9e20-df501ea5b195 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:47.432360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 913/1805] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:47 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:47.437586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e0bfd2-b139-4104-976b-604f41d952e2 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:47.443151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c2da93-b12f-4a8f-9e20-df501ea5b195 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:47.452857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39399f3-b4e2-4838-b787-07499db0fe95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:47.453586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39399f3-b4e2-4838-b787-07499db0fe95 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.453870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39399f3-b4e2-4838-b787-07499db0fe95 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.454153 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39399f3-b4e2-4838-b787-07499db0fe95 admin admin] PATH_INFO: `/v3/projects/c4d51c53bb6340f88d0b9e0614a2e51f/users/5f1fa87a36e54a9c87ec780f68da2e5d/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.457859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39399f3-b4e2-4838-b787-07499db0fe95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4d51c53bb6340f88d0b9e0614a2e51f, user_id=5f1fa87a36e54a9c87ec780f68da2e5d, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:47.463904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39399f3-b4e2-4838-b787-07499db0fe95 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:47.471491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e0bfd2-b139-4104-976b-604f41d952e2 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:47.480829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e090f29a-8096-4626-9b52-eede195dfb3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:47.481487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e090f29a-8096-4626-9b52-eede195dfb3c admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.482004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e090f29a-8096-4626-9b52-eede195dfb3c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.482481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e090f29a-8096-4626-9b52-eede195dfb3c admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.483761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e090f29a-8096-4626-9b52-eede195dfb3c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:47.487467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 893/1806] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:47 2022] PUT /identity/v3/projects/c4d51c53bb6340f88d0b9e0614a2e51f/users/5f1fa87a36e54a9c87ec780f68da2e5d/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:47.487773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e090f29a-8096-4626-9b52-eede195dfb3c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:47.505251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98811e5-19a0-4121-b0e5-b5809b573e76 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98811e5-19a0-4121-b0e5-b5809b573e76 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.506464 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98811e5-19a0-4121-b0e5-b5809b573e76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.530434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:47.544258 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c98811e5-19a0-4121-b0e5-b5809b573e76 None None] MFA Rules not processed for user `5f1fa87a36e54a9c87ec780f68da2e5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:47.545601 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c98811e5-19a0-4121-b0e5-b5809b573e76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:47.552040 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 914/1807] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:47 2022] POST /identity/v3/projects => generated 387 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:47.553874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e090f29a-8096-4626-9b52-eede195dfb3c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:47.639545 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e090f29a-8096-4626-9b52-eede195dfb3c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:47.647476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1af1f2f-24d3-4653-87b5-0e3ffae62a98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:47.648481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1af1f2f-24d3-4653-87b5-0e3ffae62a98 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.648967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1af1f2f-24d3-4653-87b5-0e3ffae62a98 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.649395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1af1f2f-24d3-4653-87b5-0e3ffae62a98 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.650535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1af1f2f-24d3-4653-87b5-0e3ffae62a98 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:47.650869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 894/1808] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:47 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:47.654210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1af1f2f-24d3-4653-87b5-0e3ffae62a98 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:47.666810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98811e5-19a0-4121-b0e5-b5809b573e76 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:47.668727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c98811e5-19a0-4121-b0e5-b5809b573e76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:47.685205 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 915/1809] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:47 2022] POST /identity/v3/users => generated 300 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:47.690186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1af1f2f-24d3-4653-87b5-0e3ffae62a98 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:47.704121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1af1f2f-24d3-4653-87b5-0e3ffae62a98 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:47.712609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8160b66b-4f6b-486c-a693-cd43df789ed6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:47.713737 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8160b66b-4f6b-486c-a693-cd43df789ed6 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.714291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8160b66b-4f6b-486c-a693-cd43df789ed6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.714721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8160b66b-4f6b-486c-a693-cd43df789ed6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.716067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8160b66b-4f6b-486c-a693-cd43df789ed6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:47.718403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8160b66b-4f6b-486c-a693-cd43df789ed6 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:47.738481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 916/1810] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:47 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:47.743052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8160b66b-4f6b-486c-a693-cd43df789ed6 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:47.751760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98811e5-19a0-4121-b0e5-b5809b573e76 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:47.756032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548f0497-d1f7-443f-abb9-7bf90d25aa2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:47.756710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548f0497-d1f7-443f-abb9-7bf90d25aa2f service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.756958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548f0497-d1f7-443f-abb9-7bf90d25aa2f service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.757257 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548f0497-d1f7-443f-abb9-7bf90d25aa2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.761596 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8160b66b-4f6b-486c-a693-cd43df789ed6 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:47.770147 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079a1366-f194-4404-a068-d5678ff2c70b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:47.770666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548f0497-d1f7-443f-abb9-7bf90d25aa2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:47.771295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079a1366-f194-4404-a068-d5678ff2c70b admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:47.771836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079a1366-f194-4404-a068-d5678ff2c70b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:47.772340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079a1366-f194-4404-a068-d5678ff2c70b admin admin] PATH_INFO: `/v3/projects/10b402ac670747e2aee749df659e323b/users/80010f8f4bb04c43984becca1598f346/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:47.772903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548f0497-d1f7-443f-abb9-7bf90d25aa2f service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:47.797561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079a1366-f194-4404-a068-d5678ff2c70b admin admin] RBAC: Authorizing `identity:create_grant(project_id=10b402ac670747e2aee749df659e323b, user_id=80010f8f4bb04c43984becca1598f346, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:47.798407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 895/1811] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:47 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:08:47.800861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079a1366-f194-4404-a068-d5678ff2c70b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:47.819273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 917/1812] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:47 2022] PUT /identity/v3/projects/10b402ac670747e2aee749df659e323b/users/80010f8f4bb04c43984becca1598f346/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:47.826152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548f0497-d1f7-443f-abb9-7bf90d25aa2f service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:47.845986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548f0497-d1f7-443f-abb9-7bf90d25aa2f service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:48.004043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84e787c-9866-42c8-b51b-457b3868c6f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:48.005152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84e787c-9866-42c8-b51b-457b3868c6f7 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:48.005678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84e787c-9866-42c8-b51b-457b3868c6f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:48.006223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84e787c-9866-42c8-b51b-457b3868c6f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:48.007433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84e787c-9866-42c8-b51b-457b3868c6f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:48.009537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84e787c-9866-42c8-b51b-457b3868c6f7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:48.035307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 896/1813] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:47 2022] GET /identity/v3/roles => generated 1420 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:48.042115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079a1366-f194-4404-a068-d5678ff2c70b admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:48.060639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079a1366-f194-4404-a068-d5678ff2c70b admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:48.070304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13f7ee8-16af-4ed9-bf0d-019fae49b77f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:48.072002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13f7ee8-16af-4ed9-bf0d-019fae49b77f admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:48.072612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13f7ee8-16af-4ed9-bf0d-019fae49b77f admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:48.073205 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13f7ee8-16af-4ed9-bf0d-019fae49b77f admin admin] PATH_INFO: `/v3/projects/10b402ac670747e2aee749df659e323b/users/80010f8f4bb04c43984becca1598f346/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:48.078564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13f7ee8-16af-4ed9-bf0d-019fae49b77f admin admin] RBAC: Authorizing `identity:create_grant(project_id=10b402ac670747e2aee749df659e323b, user_id=80010f8f4bb04c43984becca1598f346, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:48.085211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13f7ee8-16af-4ed9-bf0d-019fae49b77f admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:48.090642 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:48.105059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 918/1814] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:48 2022] PUT /identity/v3/projects/10b402ac670747e2aee749df659e323b/users/80010f8f4bb04c43984becca1598f346/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:48.117527 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d080333-a930-4550-92cc-56b00c2eba31 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:48.117990 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d080333-a930-4550-92cc-56b00c2eba31 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:48.118433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d080333-a930-4550-92cc-56b00c2eba31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:48.163170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6d080333-a930-4550-92cc-56b00c2eba31 None None] MFA Rules not processed for user `80010f8f4bb04c43984becca1598f346`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:48.164959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6d080333-a930-4550-92cc-56b00c2eba31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:48.206013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:48.299117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 897/1815] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:48 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:48.316013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13f7ee8-16af-4ed9-bf0d-019fae49b77f admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:48.398415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13f7ee8-16af-4ed9-bf0d-019fae49b77f admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:48.402164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945d1cbe-109e-473d-949f-c7e167be287b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:48.402795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945d1cbe-109e-473d-949f-c7e167be287b service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:48.403075 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945d1cbe-109e-473d-949f-c7e167be287b service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:48.403342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945d1cbe-109e-473d-949f-c7e167be287b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:48.413418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945d1cbe-109e-473d-949f-c7e167be287b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:48.415008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945d1cbe-109e-473d-949f-c7e167be287b service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:48.429886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 919/1816] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:48 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:48.467783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d080333-a930-4550-92cc-56b00c2eba31 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:48.505777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945d1cbe-109e-473d-949f-c7e167be287b service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:48.572671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d080333-a930-4550-92cc-56b00c2eba31 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:48.578067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a9dc36-57ea-43fa-99dc-352a18030089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:48.578840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a9dc36-57ea-43fa-99dc-352a18030089 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:48.579247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a9dc36-57ea-43fa-99dc-352a18030089 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:48.579683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a9dc36-57ea-43fa-99dc-352a18030089 admin admin] PATH_INFO: `/v3/users/b246c19664754525af7b2e4637257848` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:48.581140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a9dc36-57ea-43fa-99dc-352a18030089 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b246c19664754525af7b2e4637257848)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:48.583770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a9dc36-57ea-43fa-99dc-352a18030089 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:48.610075 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945d1cbe-109e-473d-949f-c7e167be287b service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:48.616406 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca706424-43b7-4c19-9d43-b706efdca2ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:48.617232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca706424-43b7-4c19-9d43-b706efdca2ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:48.619673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca706424-43b7-4c19-9d43-b706efdca2ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:48.620076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca706424-43b7-4c19-9d43-b706efdca2ca admin admin] PATH_INFO: `/v3/users/af9ed1804b254e899eb5d7fbe4840411` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:48.621719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca706424-43b7-4c19-9d43-b706efdca2ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=af9ed1804b254e899eb5d7fbe4840411)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:48.624930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca706424-43b7-4c19-9d43-b706efdca2ca admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:48.664081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-18a9dc36-57ea-43fa-99dc-352a18030089 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b246c19664754525af7b2e4637257848'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:48.675463 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-18a9dc36-57ea-43fa-99dc-352a18030089 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b246c19664754525af7b2e4637257848'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:48.676609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-18a9dc36-57ea-43fa-99dc-352a18030089 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b246c19664754525af7b2e4637257848'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:48.696320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-18a9dc36-57ea-43fa-99dc-352a18030089 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b246c19664754525af7b2e4637257848'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:48.697172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-18a9dc36-57ea-43fa-99dc-352a18030089 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b246c19664754525af7b2e4637257848'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:48.701815 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ca706424-43b7-4c19-9d43-b706efdca2ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af9ed1804b254e899eb5d7fbe4840411'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:48.702822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ca706424-43b7-4c19-9d43-b706efdca2ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af9ed1804b254e899eb5d7fbe4840411'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:48.714859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ca706424-43b7-4c19-9d43-b706efdca2ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af9ed1804b254e899eb5d7fbe4840411'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:48.718137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 898/1817] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:48 2022] DELETE /identity/v3/users/b246c19664754525af7b2e4637257848 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:48.724337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3017b47-58de-40d3-a392-d13142e5da8b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:48.724770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3017b47-58de-40d3-a392-d13142e5da8b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:48.725219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3017b47-58de-40d3-a392-d13142e5da8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:48.736570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ca706424-43b7-4c19-9d43-b706efdca2ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af9ed1804b254e899eb5d7fbe4840411'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:48.755194 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ca706424-43b7-4c19-9d43-b706efdca2ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af9ed1804b254e899eb5d7fbe4840411'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:48.756057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b3017b47-58de-40d3-a392-d13142e5da8b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:48.758051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b3017b47-58de-40d3-a392-d13142e5da8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:48.765482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 920/1818] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:48 2022] DELETE /identity/v3/users/af9ed1804b254e899eb5d7fbe4840411 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:48.772252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46ccc52-376c-49f5-a826-cced0ed9cd26 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:48.772703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46ccc52-376c-49f5-a826-cced0ed9cd26 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:48.773143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46ccc52-376c-49f5-a826-cced0ed9cd26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:48.808187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b46ccc52-376c-49f5-a826-cced0ed9cd26 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:48.809723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b46ccc52-376c-49f5-a826-cced0ed9cd26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:48.822313 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:48.831232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:48.867950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 899/1819] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:48.893583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3017b47-58de-40d3-a392-d13142e5da8b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:48.896136 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b3017b47-58de-40d3-a392-d13142e5da8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:48.942424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 921/1820] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:48.955460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46ccc52-376c-49f5-a826-cced0ed9cd26 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:48.957182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b46ccc52-376c-49f5-a826-cced0ed9cd26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:49.037920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3017b47-58de-40d3-a392-d13142e5da8b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:49.139185 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46ccc52-376c-49f5-a826-cced0ed9cd26 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:49.162184 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3934d3e1-c209-4aae-b7f7-9cb1689799cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:49.163280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3934d3e1-c209-4aae-b7f7-9cb1689799cb service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:49.163846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3934d3e1-c209-4aae-b7f7-9cb1689799cb service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:49.164422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3934d3e1-c209-4aae-b7f7-9cb1689799cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:49.194289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aad4f2c-d90b-4230-93e7-d9815f702946 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:49.195673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aad4f2c-d90b-4230-93e7-d9815f702946 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:49.197085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aad4f2c-d90b-4230-93e7-d9815f702946 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:49.199827 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aad4f2c-d90b-4230-93e7-d9815f702946 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:49.207808 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3934d3e1-c209-4aae-b7f7-9cb1689799cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:49.211402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3934d3e1-c209-4aae-b7f7-9cb1689799cb service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:49.229142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aad4f2c-d90b-4230-93e7-d9815f702946 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:49.231620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 922/1821] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:49.231916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aad4f2c-d90b-4230-93e7-d9815f702946 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:49.251090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 900/1822] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:49.680826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3934d3e1-c209-4aae-b7f7-9cb1689799cb service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:49.685621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3934d3e1-c209-4aae-b7f7-9cb1689799cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:49.685889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aad4f2c-d90b-4230-93e7-d9815f702946 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:49.691859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-3aad4f2c-d90b-4230-93e7-d9815f702946 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:49.720264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3934d3e1-c209-4aae-b7f7-9cb1689799cb service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:49.726638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:49.727500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:49.727793 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:49.728069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] PATH_INFO: `/v3/projects/abb3207f94ac48b7be28ae83b234521b` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:49.729683 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] RBAC: Authorizing `identity:delete_project(project_id=abb3207f94ac48b7be28ae83b234521b)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:49.731729 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:49.731967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aad4f2c-d90b-4230-93e7-d9815f702946 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:49.744239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:49.744935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:49.745186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:49.745465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] PATH_INFO: `/v3/projects/eac61647e4ee4a9699be0cd8f0be0264` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:49.746805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] RBAC: Authorizing `identity:delete_project(project_id=eac61647e4ee4a9699be0cd8f0be0264)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:49.760011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:49.804048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'abb3207f94ac48b7be28ae83b234521b', 'request_id': 'req-79b4794f-4e15-405f-af85-7da01ed4d095'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:49.825884 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'abb3207f94ac48b7be28ae83b234521b', 'request_id': 'req-79b4794f-4e15-405f-af85-7da01ed4d095'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:49.833737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eac61647e4ee4a9699be0cd8f0be0264', 'request_id': 'req-39e2fda0-0757-430c-9fb4-66174e872bfc'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:49.835821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'abb3207f94ac48b7be28ae83b234521b', 'request_id': 'req-79b4794f-4e15-405f-af85-7da01ed4d095'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:49.841730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eac61647e4ee4a9699be0cd8f0be0264', 'request_id': 'req-39e2fda0-0757-430c-9fb4-66174e872bfc'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:49.849411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eac61647e4ee4a9699be0cd8f0be0264', 'request_id': 'req-39e2fda0-0757-430c-9fb4-66174e872bfc'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:49.849942 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] The token cache is being invalidate because project abb3207f94ac48b7be28ae83b234521b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:49.850289 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:49.850978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:49.853192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 923/1823] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:49 2022] DELETE /identity/v3/projects/abb3207f94ac48b7be28ae83b234521b => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:49.859631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:49.867059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:49.870077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] The token cache is being invalidate because project eac61647e4ee4a9699be0cd8f0be0264 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:49.870375 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:49.871101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:49.875905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 901/1824] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:49 2022] DELETE /identity/v3/projects/eac61647e4ee4a9699be0cd8f0be0264 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:49.878011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:49.879992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:49.946540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:49.951429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:49.985238 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b4794f-4e15-405f-af85-7da01ed4d095 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:49.998951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e2fda0-0757-430c-9fb4-66174e872bfc admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:50.004641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:50.005339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:50.005617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:50.005892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] PATH_INFO: `/v3/users/80010f8f4bb04c43984becca1598f346` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:50.007213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] RBAC: Authorizing `identity:delete_user(user_id=80010f8f4bb04c43984becca1598f346)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:50.009252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:50.073757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80010f8f4bb04c43984becca1598f346'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:50.087903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80010f8f4bb04c43984becca1598f346'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:50.088828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80010f8f4bb04c43984becca1598f346'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:50.088945 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c874d9b5-3a2f-4ae1-90dc-44a7c098e9ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:50.089958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c874d9b5-3a2f-4ae1-90dc-44a7c098e9ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:50.090695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c874d9b5-3a2f-4ae1-90dc-44a7c098e9ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:50.091164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c874d9b5-3a2f-4ae1-90dc-44a7c098e9ac admin admin] PATH_INFO: `/v3/users/5f1fa87a36e54a9c87ec780f68da2e5d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:50.093769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c874d9b5-3a2f-4ae1-90dc-44a7c098e9ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f1fa87a36e54a9c87ec780f68da2e5d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:50.097694 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c874d9b5-3a2f-4ae1-90dc-44a7c098e9ac admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:50.107562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80010f8f4bb04c43984becca1598f346'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:50.108862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80010f8f4bb04c43984becca1598f346'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:50.132718 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 902/1825] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:49 2022] DELETE /identity/v3/users/80010f8f4bb04c43984becca1598f346 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:50.206894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-c874d9b5-3a2f-4ae1-90dc-44a7c098e9ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f1fa87a36e54a9c87ec780f68da2e5d'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:50.208277 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-c874d9b5-3a2f-4ae1-90dc-44a7c098e9ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f1fa87a36e54a9c87ec780f68da2e5d'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:50.225837 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-c874d9b5-3a2f-4ae1-90dc-44a7c098e9ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f1fa87a36e54a9c87ec780f68da2e5d'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:50.264097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-c874d9b5-3a2f-4ae1-90dc-44a7c098e9ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f1fa87a36e54a9c87ec780f68da2e5d'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:50.283957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-c874d9b5-3a2f-4ae1-90dc-44a7c098e9ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f1fa87a36e54a9c87ec780f68da2e5d'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:50.317294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 924/1826] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:49 2022] DELETE /identity/v3/users/5f1fa87a36e54a9c87ec780f68da2e5d => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:51.991230 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:51.993554 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:52.234911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e154f24-50ed-4723-a33d-7bd42f46cd3e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:52.445235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:52.498466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0699e57-5a40-479b-afa2-8df3ce6490fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:52.499935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0699e57-5a40-479b-afa2-8df3ce6490fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:52.500402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0699e57-5a40-479b-afa2-8df3ce6490fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:52.504550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0699e57-5a40-479b-afa2-8df3ce6490fc admin admin] PATH_INFO: `/v3/projects/10b402ac670747e2aee749df659e323b` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:52.507087 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0699e57-5a40-479b-afa2-8df3ce6490fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=10b402ac670747e2aee749df659e323b)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:52.509788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0699e57-5a40-479b-afa2-8df3ce6490fc admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:52.670256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-b0699e57-5a40-479b-afa2-8df3ce6490fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '10b402ac670747e2aee749df659e323b', 'request_id': 'req-b0699e57-5a40-479b-afa2-8df3ce6490fc'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:52.681377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-b0699e57-5a40-479b-afa2-8df3ce6490fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '10b402ac670747e2aee749df659e323b', 'request_id': 'req-b0699e57-5a40-479b-afa2-8df3ce6490fc'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:52.692049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-b0699e57-5a40-479b-afa2-8df3ce6490fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '10b402ac670747e2aee749df659e323b', 'request_id': 'req-b0699e57-5a40-479b-afa2-8df3ce6490fc'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:52.725227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-b0699e57-5a40-479b-afa2-8df3ce6490fc admin admin] The token cache is being invalidate because project 10b402ac670747e2aee749df659e323b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:52.726161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-b0699e57-5a40-479b-afa2-8df3ce6490fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:52.727871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-b0699e57-5a40-479b-afa2-8df3ce6490fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:52.737870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 903/1827] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:51 2022] DELETE /identity/v3/projects/10b402ac670747e2aee749df659e323b => generated 0 bytes in 742 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:52.739886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f210510a-5cb3-4911-9284-52789f49d4c2 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:52.740692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f210510a-5cb3-4911-9284-52789f49d4c2 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:52.740692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f210510a-5cb3-4911-9284-52789f49d4c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:52.777932 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f210510a-5cb3-4911-9284-52789f49d4c2 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:52.781028 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f210510a-5cb3-4911-9284-52789f49d4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:52.911139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 925/1828] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:52.919499 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c69889-533c-457c-b2bf-44c4145128e9 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:52.919955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c69889-533c-457c-b2bf-44c4145128e9 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:52.920306 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c69889-533c-457c-b2bf-44c4145128e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:52.944410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-50c69889-533c-457c-b2bf-44c4145128e9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:52.945736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-50c69889-533c-457c-b2bf-44c4145128e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:52.955827 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 904/1829] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:52.960052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f210510a-5cb3-4911-9284-52789f49d4c2 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:52.973025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f210510a-5cb3-4911-9284-52789f49d4c2 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:52.978467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dc8594-5eaf-42d2-916e-a0df681ad9ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:52.979612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dc8594-5eaf-42d2-916e-a0df681ad9ee admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:52.980077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dc8594-5eaf-42d2-916e-a0df681ad9ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:52.981263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dc8594-5eaf-42d2-916e-a0df681ad9ee admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:52.982352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95dc8594-5eaf-42d2-916e-a0df681ad9ee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:08:52.982887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95dc8594-5eaf-42d2-916e-a0df681ad9ee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:52.984809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95dc8594-5eaf-42d2-916e-a0df681ad9ee admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:52.993659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:53.005901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c69889-533c-457c-b2bf-44c4145128e9 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:53.012647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 926/1830] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:08:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:08:53.019600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c69889-533c-457c-b2bf-44c4145128e9 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:53.028498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668f190a-f8b1-4d3d-a8be-df578e789247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:53.029377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668f190a-f8b1-4d3d-a8be-df578e789247 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:53.029669 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668f190a-f8b1-4d3d-a8be-df578e789247 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:53.029936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668f190a-f8b1-4d3d-a8be-df578e789247 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:53.030634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668f190a-f8b1-4d3d-a8be-df578e789247 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:53.033403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668f190a-f8b1-4d3d-a8be-df578e789247 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:53.091879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 905/1831] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:53 2022] POST /identity/v3/projects => generated 379 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:53.096535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dc8594-5eaf-42d2-916e-a0df681ad9ee admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:53.130389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dc8594-5eaf-42d2-916e-a0df681ad9ee admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:53.225161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668f190a-f8b1-4d3d-a8be-df578e789247 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:53.227155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-668f190a-f8b1-4d3d-a8be-df578e789247 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:53.255710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad880503-548d-4558-b628-d3bd511635ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:53.255710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad880503-548d-4558-b628-d3bd511635ba admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:53.255710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad880503-548d-4558-b628-d3bd511635ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:53.255710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad880503-548d-4558-b628-d3bd511635ba admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:53.256018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad880503-548d-4558-b628-d3bd511635ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:53.256018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad880503-548d-4558-b628-d3bd511635ba admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:53.319160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 927/1832] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:53 2022] POST /identity/v3/users => generated 290 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:53.322777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad880503-548d-4558-b628-d3bd511635ba admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:53.358315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad880503-548d-4558-b628-d3bd511635ba admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:53.384226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2b62e4-9b36-4c03-944d-68d12fa4f54b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:53.384962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2b62e4-9b36-4c03-944d-68d12fa4f54b admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:53.385246 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2b62e4-9b36-4c03-944d-68d12fa4f54b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:53.385547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2b62e4-9b36-4c03-944d-68d12fa4f54b admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:53.386345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2b62e4-9b36-4c03-944d-68d12fa4f54b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:53.387867 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668f190a-f8b1-4d3d-a8be-df578e789247 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:53.391709 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2b62e4-9b36-4c03-944d-68d12fa4f54b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:53.404519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:53.405294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:53.405654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:53.406017 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3 admin admin] PATH_INFO: `/v3/projects/c4d51c53bb6340f88d0b9e0614a2e51f` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:53.407497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4d51c53bb6340f88d0b9e0614a2e51f)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:53.409576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 928/1833] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:53 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:53.420726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:53.421242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2b62e4-9b36-4c03-944d-68d12fa4f54b admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:53.442510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2b62e4-9b36-4c03-944d-68d12fa4f54b admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:53.455491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143a3260-a63d-4654-a03c-3e0f2e8ba121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:53.455920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143a3260-a63d-4654-a03c-3e0f2e8ba121 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:53.456625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143a3260-a63d-4654-a03c-3e0f2e8ba121 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:53.457163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143a3260-a63d-4654-a03c-3e0f2e8ba121 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:53.458145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143a3260-a63d-4654-a03c-3e0f2e8ba121 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:53.459855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143a3260-a63d-4654-a03c-3e0f2e8ba121 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:53.488334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 929/1834] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:53 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:53.490841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4d51c53bb6340f88d0b9e0614a2e51f', 'request_id': 'req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:53.491779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143a3260-a63d-4654-a03c-3e0f2e8ba121 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:53.494076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:53.500345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4d51c53bb6340f88d0b9e0614a2e51f', 'request_id': 'req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:53.513282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143a3260-a63d-4654-a03c-3e0f2e8ba121 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:53.520771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4d51c53bb6340f88d0b9e0614a2e51f', 'request_id': 'req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:53.557558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3 admin admin] The token cache is being invalidate because project c4d51c53bb6340f88d0b9e0614a2e51f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:53.557909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:53.564292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-cf4f0d22-1a25-44b3-acf2-245dc9cfb6b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:53.567671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 906/1835] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:53 2022] DELETE /identity/v3/projects/c4d51c53bb6340f88d0b9e0614a2e51f => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:53.576729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999db768-4aaa-45f2-aeae-bbe2c9f23cde None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:53.577043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999db768-4aaa-45f2-aeae-bbe2c9f23cde None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:53.577362 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999db768-4aaa-45f2-aeae-bbe2c9f23cde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:53.609107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-999db768-4aaa-45f2-aeae-bbe2c9f23cde None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:53.611301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-999db768-4aaa-45f2-aeae-bbe2c9f23cde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:53.618915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7552bdee-eda2-4f17-ba3e-deb14d233519 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:53.620054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7552bdee-eda2-4f17-ba3e-deb14d233519 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:53.620682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7552bdee-eda2-4f17-ba3e-deb14d233519 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:53.621010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7552bdee-eda2-4f17-ba3e-deb14d233519 admin admin] PATH_INFO: `/v3/projects/3c9901d2ab45408f9662f17986097fe9/users/ecc4c980fe9c43ad9998e93ab4b0c32f/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:53.638488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7552bdee-eda2-4f17-ba3e-deb14d233519 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c9901d2ab45408f9662f17986097fe9, user_id=ecc4c980fe9c43ad9998e93ab4b0c32f, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:53.641851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7552bdee-eda2-4f17-ba3e-deb14d233519 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:53.661564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 930/1836] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:53 2022] PUT /identity/v3/projects/3c9901d2ab45408f9662f17986097fe9/users/ecc4c980fe9c43ad9998e93ab4b0c32f/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:53.677206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4919bed-f027-491a-b185-557317fbe44d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:53.678003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4919bed-f027-491a-b185-557317fbe44d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:53.681666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4919bed-f027-491a-b185-557317fbe44d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:53.727412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 907/1837] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:53.741743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b4919bed-f027-491a-b185-557317fbe44d None None] MFA Rules not processed for user `ecc4c980fe9c43ad9998e93ab4b0c32f`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:53.742272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f39234-5b33-45f8-9669-eef120cb24b9 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:53.742272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f39234-5b33-45f8-9669-eef120cb24b9 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:53.742692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f39234-5b33-45f8-9669-eef120cb24b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:53.743037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b4919bed-f027-491a-b185-557317fbe44d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:53.789883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a2f39234-5b33-45f8-9669-eef120cb24b9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:53.792905 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:53.793951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a2f39234-5b33-45f8-9669-eef120cb24b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:53.849493 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 931/1838] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:53 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:08:53.854779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4919bed-f027-491a-b185-557317fbe44d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:53.855010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b4919bed-f027-491a-b185-557317fbe44d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:53.911332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 908/1839] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:53.917712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f39234-5b33-45f8-9669-eef120cb24b9 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:53.929849 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f39234-5b33-45f8-9669-eef120cb24b9 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:53.940378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9539393f-0c2b-4b23-b027-57c5202dad75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:53.942387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9539393f-0c2b-4b23-b027-57c5202dad75 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:53.943041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9539393f-0c2b-4b23-b027-57c5202dad75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:53.943304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4919bed-f027-491a-b185-557317fbe44d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:53.943803 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9539393f-0c2b-4b23-b027-57c5202dad75 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:53.946740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9539393f-0c2b-4b23-b027-57c5202dad75 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:08:53.947425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9539393f-0c2b-4b23-b027-57c5202dad75 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:53.948393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9c9241-89f7-4465-8872-4039ed1a5351 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:53.949031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9c9241-89f7-4465-8872-4039ed1a5351 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:53.949315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9c9241-89f7-4465-8872-4039ed1a5351 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:53.949591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9c9241-89f7-4465-8872-4039ed1a5351 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:53.950439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9c9241-89f7-4465-8872-4039ed1a5351 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:53.952670 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9539393f-0c2b-4b23-b027-57c5202dad75 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:53.953098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9c9241-89f7-4465-8872-4039ed1a5351 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:53.976663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 909/1840] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:08:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:08:53.982157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9539393f-0c2b-4b23-b027-57c5202dad75 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:53.998993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9539393f-0c2b-4b23-b027-57c5202dad75 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.005071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a372ec81-8ac3-494e-ba2e-74e6a75e5fdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.005721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a372ec81-8ac3-494e-ba2e-74e6a75e5fdf admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.006056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a372ec81-8ac3-494e-ba2e-74e6a75e5fdf admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.006903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a372ec81-8ac3-494e-ba2e-74e6a75e5fdf admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.007058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a372ec81-8ac3-494e-ba2e-74e6a75e5fdf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.009298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a372ec81-8ac3-494e-ba2e-74e6a75e5fdf admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:54.019146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 932/1841] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:53 2022] POST /identity/v3/projects => generated 379 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:54.023947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9c9241-89f7-4465-8872-4039ed1a5351 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.060751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 910/1842] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:53 2022] POST /identity/v3/projects => generated 379 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:54.072474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a372ec81-8ac3-494e-ba2e-74e6a75e5fdf admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.138141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9c9241-89f7-4465-8872-4039ed1a5351 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.177729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:DELETE=3,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:54.209411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a372ec81-8ac3-494e-ba2e-74e6a75e5fdf admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.216308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d808f57-fb2c-4cbb-be5d-7f855f7ed7a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.217307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d808f57-fb2c-4cbb-be5d-7f855f7ed7a5 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.217618 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d808f57-fb2c-4cbb-be5d-7f855f7ed7a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.218006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d808f57-fb2c-4cbb-be5d-7f855f7ed7a5 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.218770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d808f57-fb2c-4cbb-be5d-7f855f7ed7a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.221508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d808f57-fb2c-4cbb-be5d-7f855f7ed7a5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:54.256006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 911/1843] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:54 2022] POST /identity/v3/users => generated 290 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:54.258401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d808f57-fb2c-4cbb-be5d-7f855f7ed7a5 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.261402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93540316-d05d-43ab-8460-b7df1880325d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.262052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93540316-d05d-43ab-8460-b7df1880325d admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.262334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93540316-d05d-43ab-8460-b7df1880325d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.262609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93540316-d05d-43ab-8460-b7df1880325d admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.264045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93540316-d05d-43ab-8460-b7df1880325d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.267615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93540316-d05d-43ab-8460-b7df1880325d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:54.274332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d808f57-fb2c-4cbb-be5d-7f855f7ed7a5 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.283412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b6f46d-1449-44bf-85ea-8428485340be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.284636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b6f46d-1449-44bf-85ea-8428485340be admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.285088 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b6f46d-1449-44bf-85ea-8428485340be admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.285544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b6f46d-1449-44bf-85ea-8428485340be admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.286455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b6f46d-1449-44bf-85ea-8428485340be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.289341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b6f46d-1449-44bf-85ea-8428485340be admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:54.307105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 912/1844] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:54.311153 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b6f46d-1449-44bf-85ea-8428485340be admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.317494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:54.331743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 933/1845] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:54 2022] POST /identity/v3/users => generated 290 bytes in 309 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:54.332474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b6f46d-1449-44bf-85ea-8428485340be admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.332885 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93540316-d05d-43ab-8460-b7df1880325d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.338237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cdcddd-3e44-47bb-b3ab-e6eee8b966b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.339291 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cdcddd-3e44-47bb-b3ab-e6eee8b966b6 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.339671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cdcddd-3e44-47bb-b3ab-e6eee8b966b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.340053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cdcddd-3e44-47bb-b3ab-e6eee8b966b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.340964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cdcddd-3e44-47bb-b3ab-e6eee8b966b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.342891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cdcddd-3e44-47bb-b3ab-e6eee8b966b6 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:54.346423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93540316-d05d-43ab-8460-b7df1880325d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.352671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a582eed6-2ddb-4668-8d0e-1a8db9fb081f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.353331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a582eed6-2ddb-4668-8d0e-1a8db9fb081f admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.353567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a582eed6-2ddb-4668-8d0e-1a8db9fb081f admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.353899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a582eed6-2ddb-4668-8d0e-1a8db9fb081f admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.354629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a582eed6-2ddb-4668-8d0e-1a8db9fb081f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.356514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a582eed6-2ddb-4668-8d0e-1a8db9fb081f admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:54.364846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 913/1846] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:54.365817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cdcddd-3e44-47bb-b3ab-e6eee8b966b6 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.375239 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 934/1847] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:54.376310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a582eed6-2ddb-4668-8d0e-1a8db9fb081f admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.384258 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cdcddd-3e44-47bb-b3ab-e6eee8b966b6 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.389200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a582eed6-2ddb-4668-8d0e-1a8db9fb081f admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.390121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2570911-3cf1-40cd-bf35-5431917a2bcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.390957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2570911-3cf1-40cd-bf35-5431917a2bcd admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.391320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2570911-3cf1-40cd-bf35-5431917a2bcd admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.392320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2570911-3cf1-40cd-bf35-5431917a2bcd admin admin] PATH_INFO: `/v3/projects/926a1fb02a7c435b9e1ec997271536e2/users/e8dd5929d2844bd9bb1e1d52a249d0f4/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.396284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a86d08-20e2-4482-b75c-b392c47c8855 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.396992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a86d08-20e2-4482-b75c-b392c47c8855 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.397274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a86d08-20e2-4482-b75c-b392c47c8855 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.397549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a86d08-20e2-4482-b75c-b392c47c8855 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.398265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a86d08-20e2-4482-b75c-b392c47c8855 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.400234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a86d08-20e2-4482-b75c-b392c47c8855 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:54.414846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 935/1848] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:54.419000 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a86d08-20e2-4482-b75c-b392c47c8855 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.419719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2570911-3cf1-40cd-bf35-5431917a2bcd admin admin] RBAC: Authorizing `identity:create_grant(project_id=926a1fb02a7c435b9e1ec997271536e2, user_id=e8dd5929d2844bd9bb1e1d52a249d0f4, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.423386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2570911-3cf1-40cd-bf35-5431917a2bcd admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:54.437994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a86d08-20e2-4482-b75c-b392c47c8855 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.441659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 914/1849] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:54 2022] PUT /identity/v3/projects/926a1fb02a7c435b9e1ec997271536e2/users/e8dd5929d2844bd9bb1e1d52a249d0f4/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:54.452517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10513f57-91a3-4c56-816e-61c60fd90675 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.452517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10513f57-91a3-4c56-816e-61c60fd90675 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10513f57-91a3-4c56-816e-61c60fd90675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.488288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-10513f57-91a3-4c56-816e-61c60fd90675 None None] MFA Rules not processed for user `e8dd5929d2844bd9bb1e1d52a249d0f4`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:54.489531 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-10513f57-91a3-4c56-816e-61c60fd90675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:54.509202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04da158-c618-4836-a4e7-f0257cef11b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.509598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04da158-c618-4836-a4e7-f0257cef11b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.509873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04da158-c618-4836-a4e7-f0257cef11b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.510155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04da158-c618-4836-a4e7-f0257cef11b4 admin admin] PATH_INFO: `/v3/projects/444d903f04c540c5ab06974e609b4923/users/cce60d9fb54144b59e7bccec438a36a4/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.531666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04da158-c618-4836-a4e7-f0257cef11b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=444d903f04c540c5ab06974e609b4923, user_id=cce60d9fb54144b59e7bccec438a36a4, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.535750 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04da158-c618-4836-a4e7-f0257cef11b4 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:54.553003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 936/1850] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:54 2022] PUT /identity/v3/projects/444d903f04c540c5ab06974e609b4923/users/cce60d9fb54144b59e7bccec438a36a4/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:54.565287 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f75b1-4307-4800-b4d2-2a14504b3730 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.565552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f75b1-4307-4800-b4d2-2a14504b3730 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.565852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f75b1-4307-4800-b4d2-2a14504b3730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.583020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 915/1851] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:54 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:54.584942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10513f57-91a3-4c56-816e-61c60fd90675 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.597547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10513f57-91a3-4c56-816e-61c60fd90675 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.611382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-cf0f75b1-4307-4800-b4d2-2a14504b3730 None None] MFA Rules not processed for user `cce60d9fb54144b59e7bccec438a36a4`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:54.612680 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-cf0f75b1-4307-4800-b4d2-2a14504b3730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:54.675637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa1a05e-5f3d-4937-af57-92cb6af0cb91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.676422 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa1a05e-5f3d-4937-af57-92cb6af0cb91 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.676794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa1a05e-5f3d-4937-af57-92cb6af0cb91 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.677131 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa1a05e-5f3d-4937-af57-92cb6af0cb91 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.677883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa1a05e-5f3d-4937-af57-92cb6af0cb91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.679925 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa1a05e-5f3d-4937-af57-92cb6af0cb91 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:54.702606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 937/1852] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:54 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:54.703767 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0f75b1-4307-4800-b4d2-2a14504b3730 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.716719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 916/1853] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:54 2022] POST /identity/v3/projects => generated 381 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:54.722839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa1a05e-5f3d-4937-af57-92cb6af0cb91 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.726255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0f75b1-4307-4800-b4d2-2a14504b3730 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.732187 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:54.795455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:54.803547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa1a05e-5f3d-4937-af57-92cb6af0cb91 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.810873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da4c6e0-4cdc-44b3-a09c-da3cf6f92328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.811209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa9f10f-987d-4b49-b8d1-c48657151d46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.811846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da4c6e0-4cdc-44b3-a09c-da3cf6f92328 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.812017 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa9f10f-987d-4b49-b8d1-c48657151d46 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.812240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa9f10f-987d-4b49-b8d1-c48657151d46 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.812396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da4c6e0-4cdc-44b3-a09c-da3cf6f92328 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.812614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa9f10f-987d-4b49-b8d1-c48657151d46 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.812795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da4c6e0-4cdc-44b3-a09c-da3cf6f92328 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.813426 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa9f10f-987d-4b49-b8d1-c48657151d46 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.814130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da4c6e0-4cdc-44b3-a09c-da3cf6f92328 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.816377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa9f10f-987d-4b49-b8d1-c48657151d46 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:54.816608 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da4c6e0-4cdc-44b3-a09c-da3cf6f92328 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:54.863376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 917/1854] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:54 2022] POST /identity/v3/users => generated 297 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:54.864843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa9f10f-987d-4b49-b8d1-c48657151d46 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.867334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 938/1855] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:54 2022] POST /identity/v3/projects => generated 379 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:08:54.873953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da4c6e0-4cdc-44b3-a09c-da3cf6f92328 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:54.882723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa9f10f-987d-4b49-b8d1-c48657151d46 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:54.987227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df8ec0e-d887-4f7f-ae98-a0e4328397ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:54.988441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df8ec0e-d887-4f7f-ae98-a0e4328397ac admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:54.988879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df8ec0e-d887-4f7f-ae98-a0e4328397ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:54.989585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df8ec0e-d887-4f7f-ae98-a0e4328397ac admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:54.991175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df8ec0e-d887-4f7f-ae98-a0e4328397ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:54.993902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df8ec0e-d887-4f7f-ae98-a0e4328397ac admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:55.011578 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 918/1856] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:55.012678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da4c6e0-4cdc-44b3-a09c-da3cf6f92328 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:55.013419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df8ec0e-d887-4f7f-ae98-a0e4328397ac admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:55.018746 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7e9c54-f122-416a-a2ac-8903b495642f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:55.019284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7e9c54-f122-416a-a2ac-8903b495642f admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.019736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7e9c54-f122-416a-a2ac-8903b495642f admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.019917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7e9c54-f122-416a-a2ac-8903b495642f admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.020619 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7e9c54-f122-416a-a2ac-8903b495642f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:55.022859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7e9c54-f122-416a-a2ac-8903b495642f admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:55.033995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df8ec0e-d887-4f7f-ae98-a0e4328397ac admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:55.043082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2394cb67-368c-4996-88ec-41b56a6bd8e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:55.044270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2394cb67-368c-4996-88ec-41b56a6bd8e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.044949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2394cb67-368c-4996-88ec-41b56a6bd8e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.045244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2394cb67-368c-4996-88ec-41b56a6bd8e8 admin admin] PATH_INFO: `/v3/projects/78a43db51b7e4c4183fee24dbc6ddaa6/users/d8405dc1f3024f97849e871164cb07ad/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.053760 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 939/1857] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:08:54 2022] POST /identity/v3/users => generated 296 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:55.057755 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7e9c54-f122-416a-a2ac-8903b495642f admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:55.069898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7e9c54-f122-416a-a2ac-8903b495642f admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:55.078463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1336d3d3-9898-4d29-bdfc-fdc16f193966 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:55.079597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1336d3d3-9898-4d29-bdfc-fdc16f193966 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.079913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1336d3d3-9898-4d29-bdfc-fdc16f193966 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.080113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2394cb67-368c-4996-88ec-41b56a6bd8e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78a43db51b7e4c4183fee24dbc6ddaa6, user_id=d8405dc1f3024f97849e871164cb07ad, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:55.080412 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1336d3d3-9898-4d29-bdfc-fdc16f193966 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.081606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1336d3d3-9898-4d29-bdfc-fdc16f193966 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:55.084951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1336d3d3-9898-4d29-bdfc-fdc16f193966 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:55.087998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2394cb67-368c-4996-88ec-41b56a6bd8e8 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:55.104801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 940/1858] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:55 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:08:55.109574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1336d3d3-9898-4d29-bdfc-fdc16f193966 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:55.115457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 919/1859] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:55 2022] PUT /identity/v3/projects/78a43db51b7e4c4183fee24dbc6ddaa6/users/d8405dc1f3024f97849e871164cb07ad/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:55.121142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2394cb67-368c-4996-88ec-41b56a6bd8e8 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:55.123175 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1336d3d3-9898-4d29-bdfc-fdc16f193966 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:55.209571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7b0028-acdd-4cde-b13e-1831347937ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:55.210392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7b0028-acdd-4cde-b13e-1831347937ce admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.210821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7b0028-acdd-4cde-b13e-1831347937ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.211128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7b0028-acdd-4cde-b13e-1831347937ce admin admin] PATH_INFO: `/v3/projects/d7f45101a3d24ee8a180b15a3298b20b/users/0b030fbbccd94f71ba645e1acb626e73/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.234273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7b0028-acdd-4cde-b13e-1831347937ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7f45101a3d24ee8a180b15a3298b20b, user_id=0b030fbbccd94f71ba645e1acb626e73, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:55.236280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7b0028-acdd-4cde-b13e-1831347937ce admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:55.239138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2394cb67-368c-4996-88ec-41b56a6bd8e8 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:55.245151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94106e1-2540-43fd-8749-4079ef23522c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:55.246094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94106e1-2540-43fd-8749-4079ef23522c admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.246583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94106e1-2540-43fd-8749-4079ef23522c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.246819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94106e1-2540-43fd-8749-4079ef23522c admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.247593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94106e1-2540-43fd-8749-4079ef23522c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:55.249235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94106e1-2540-43fd-8749-4079ef23522c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:55.260388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 941/1860] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:55 2022] PUT /identity/v3/projects/d7f45101a3d24ee8a180b15a3298b20b/users/0b030fbbccd94f71ba645e1acb626e73/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:55.262180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7b0028-acdd-4cde-b13e-1831347937ce admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:55.268941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 920/1861] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:55 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:55.272309 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94106e1-2540-43fd-8749-4079ef23522c admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:55.277766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:55.284600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94106e1-2540-43fd-8749-4079ef23522c admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:55.336728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:55.375634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7b0028-acdd-4cde-b13e-1831347937ce admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:55.384036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38437ef-fbb6-4755-b04a-74ecb56c696e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:55.385148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38437ef-fbb6-4755-b04a-74ecb56c696e admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.385768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38437ef-fbb6-4755-b04a-74ecb56c696e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.386336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38437ef-fbb6-4755-b04a-74ecb56c696e admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.387699 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38437ef-fbb6-4755-b04a-74ecb56c696e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:55.390003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea74dbc0-9754-46d0-944e-456b7a5909ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:55.390003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea74dbc0-9754-46d0-944e-456b7a5909ab admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.390343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea74dbc0-9754-46d0-944e-456b7a5909ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.392352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea74dbc0-9754-46d0-944e-456b7a5909ab admin admin] PATH_INFO: `/v3/projects/78a43db51b7e4c4183fee24dbc6ddaa6/users/d8405dc1f3024f97849e871164cb07ad/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.392571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38437ef-fbb6-4755-b04a-74ecb56c696e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:55.394523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea74dbc0-9754-46d0-944e-456b7a5909ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=78a43db51b7e4c4183fee24dbc6ddaa6, user_id=d8405dc1f3024f97849e871164cb07ad, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:55.397338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea74dbc0-9754-46d0-944e-456b7a5909ab admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:55.410586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 921/1862] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:55 2022] PUT /identity/v3/projects/78a43db51b7e4c4183fee24dbc6ddaa6/users/d8405dc1f3024f97849e871164cb07ad/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:55.412361 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 942/1863] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:08:55 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:08:55.419959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea74dbc0-9754-46d0-944e-456b7a5909ab admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:55.428148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc493e4-cc6e-404c-a445-0f0cf3f20937 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.428482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc493e4-cc6e-404c-a445-0f0cf3f20937 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.428790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc493e4-cc6e-404c-a445-0f0cf3f20937 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.492776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-edc493e4-cc6e-404c-a445-0f0cf3f20937 None None] MFA Rules not processed for user `d8405dc1f3024f97849e871164cb07ad`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:55.494466 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-edc493e4-cc6e-404c-a445-0f0cf3f20937 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:55.509482 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea74dbc0-9754-46d0-944e-456b7a5909ab admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:55.517520 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75431728-242f-466f-986c-fd6ed124d9fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:55.518689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75431728-242f-466f-986c-fd6ed124d9fb admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.518689 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75431728-242f-466f-986c-fd6ed124d9fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.518994 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75431728-242f-466f-986c-fd6ed124d9fb admin admin] PATH_INFO: `/v3/projects/d7f45101a3d24ee8a180b15a3298b20b/users/0b030fbbccd94f71ba645e1acb626e73/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.520919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75431728-242f-466f-986c-fd6ed124d9fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7f45101a3d24ee8a180b15a3298b20b, user_id=0b030fbbccd94f71ba645e1acb626e73, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:55.531191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75431728-242f-466f-986c-fd6ed124d9fb admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:55.553830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 922/1864] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:08:55 2022] PUT /identity/v3/projects/d7f45101a3d24ee8a180b15a3298b20b/users/0b030fbbccd94f71ba645e1acb626e73/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:55.566862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098fbd34-ce35-483f-939d-574cfe140330 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.567348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098fbd34-ce35-483f-939d-574cfe140330 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.567849 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098fbd34-ce35-483f-939d-574cfe140330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.598983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 943/1865] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:55 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:55.613036 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-098fbd34-ce35-483f-939d-574cfe140330 None None] MFA Rules not processed for user `0b030fbbccd94f71ba645e1acb626e73`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:55.614720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-098fbd34-ce35-483f-939d-574cfe140330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:55.619228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc493e4-cc6e-404c-a445-0f0cf3f20937 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:55.620954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-edc493e4-cc6e-404c-a445-0f0cf3f20937 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:55.714008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc493e4-cc6e-404c-a445-0f0cf3f20937 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:55.739083 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 923/1866] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:55 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:55.801033 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098fbd34-ce35-483f-939d-574cfe140330 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:55.801309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48323c1e-9c5f-4b5c-bbc3-56998767b7af service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:55.802701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48323c1e-9c5f-4b5c-bbc3-56998767b7af service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.802979 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48323c1e-9c5f-4b5c-bbc3-56998767b7af service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.803083 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-098fbd34-ce35-483f-939d-574cfe140330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:55.803379 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48323c1e-9c5f-4b5c-bbc3-56998767b7af service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.818530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098fbd34-ce35-483f-939d-574cfe140330 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:55.822538 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7768a7-0b45-42c8-9f7c-b1ceaa613ede service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:55.823166 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7768a7-0b45-42c8-9f7c-b1ceaa613ede service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.823477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7768a7-0b45-42c8-9f7c-b1ceaa613ede service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.823749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7768a7-0b45-42c8-9f7c-b1ceaa613ede service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.834804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7768a7-0b45-42c8-9f7c-b1ceaa613ede service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:55.837264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7768a7-0b45-42c8-9f7c-b1ceaa613ede service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:55.841335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:55.849123 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 924/1867] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:55 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:08:55.878056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48323c1e-9c5f-4b5c-bbc3-56998767b7af service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:55.879544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48323c1e-9c5f-4b5c-bbc3-56998767b7af service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:55.891447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 944/1868] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:55 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 273 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:55.896135 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7768a7-0b45-42c8-9f7c-b1ceaa613ede service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:55.901521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:55.908681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7768a7-0b45-42c8-9f7c-b1ceaa613ede service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:55.913415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de1db41-4720-4a60-bcc6-713c157f562a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:55.914429 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1db41-4720-4a60-bcc6-713c157f562a service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:55.914919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1db41-4720-4a60-bcc6-713c157f562a service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:55.915597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1db41-4720-4a60-bcc6-713c157f562a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:55.987191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de1db41-4720-4a60-bcc6-713c157f562a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:55.988690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de1db41-4720-4a60-bcc6-713c157f562a service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:55.999752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 925/1869] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:55 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:08:56.432818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48323c1e-9c5f-4b5c-bbc3-56998767b7af service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:56.446234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48323c1e-9c5f-4b5c-bbc3-56998767b7af service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:56.452188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5daf36f5-c7d9-4c7f-b923-80d915a664b1 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8405dc1f3024f97849e871164cb07ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '78a43db51b7e4c4183fee24dbc6ddaa6', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:56.452838 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5daf36f5-c7d9-4c7f-b923-80d915a664b1 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:56.453178 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5daf36f5-c7d9-4c7f-b923-80d915a664b1 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:56.453477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5daf36f5-c7d9-4c7f-b923-80d915a664b1 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:56.454969 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5daf36f5-c7d9-4c7f-b923-80d915a664b1 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:08:56.455320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5daf36f5-c7d9-4c7f-b923-80d915a664b1 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:56.458091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5daf36f5-c7d9-4c7f-b923-80d915a664b1 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:56.479750 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 945/1870] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:08:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:08:56.481312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de1db41-4720-4a60-bcc6-713c157f562a service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:56.503006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de1db41-4720-4a60-bcc6-713c157f562a service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:56.515530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a21707d-b1e2-41a4-b7a2-e9b1d4519981 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8405dc1f3024f97849e871164cb07ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '78a43db51b7e4c4183fee24dbc6ddaa6', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:56.516310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a21707d-b1e2-41a4-b7a2-e9b1d4519981 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:56.516669 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a21707d-b1e2-41a4-b7a2-e9b1d4519981 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:56.517468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a21707d-b1e2-41a4-b7a2-e9b1d4519981 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:56.520340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a21707d-b1e2-41a4-b7a2-e9b1d4519981 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:56.524997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a21707d-b1e2-41a4-b7a2-e9b1d4519981 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:56.602786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 926/1871] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:08:56 2022] POST /identity/v3/projects => generated 353 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:08:56.609851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5daf36f5-c7d9-4c7f-b923-80d915a664b1 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:56.651949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5daf36f5-c7d9-4c7f-b923-80d915a664b1 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:56.817845 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ef653e-ca14-42de-9c75-2b449dd21e55 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8405dc1f3024f97849e871164cb07ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '78a43db51b7e4c4183fee24dbc6ddaa6', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:56.818893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef653e-ca14-42de-9c75-2b449dd21e55 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:56.819152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef653e-ca14-42de-9c75-2b449dd21e55 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:56.819450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef653e-ca14-42de-9c75-2b449dd21e55 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:56.820817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ef653e-ca14-42de-9c75-2b449dd21e55 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:08:56.821182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ef653e-ca14-42de-9c75-2b449dd21e55 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:56.823080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ef653e-ca14-42de-9c75-2b449dd21e55 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:56.843125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 946/1872] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:08:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:08:56.981202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a21707d-b1e2-41a4-b7a2-e9b1d4519981 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:56.993267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a21707d-b1e2-41a4-b7a2-e9b1d4519981 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:57.000139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b034f2-f9fe-4911-a55b-826a4ce03283 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8405dc1f3024f97849e871164cb07ad', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '78a43db51b7e4c4183fee24dbc6ddaa6', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:57.000139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b034f2-f9fe-4911-a55b-826a4ce03283 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:57.000139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b034f2-f9fe-4911-a55b-826a4ce03283 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:57.000685 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b034f2-f9fe-4911-a55b-826a4ce03283 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] PATH_INFO: `/v3/projects/8294a789da974832918643928932b7a1` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:57.002588 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b034f2-f9fe-4911-a55b-826a4ce03283 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: Authorizing `identity:delete_project(project_id=8294a789da974832918643928932b7a1)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:57.005089 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b034f2-f9fe-4911-a55b-826a4ce03283 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:57.103267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c8b034f2-f9fe-4911-a55b-826a4ce03283 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8294a789da974832918643928932b7a1', 'request_id': 'req-c8b034f2-f9fe-4911-a55b-826a4ce03283'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:57.132285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c8b034f2-f9fe-4911-a55b-826a4ce03283 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8294a789da974832918643928932b7a1', 'request_id': 'req-c8b034f2-f9fe-4911-a55b-826a4ce03283'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:57.148314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c8b034f2-f9fe-4911-a55b-826a4ce03283 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8294a789da974832918643928932b7a1', 'request_id': 'req-c8b034f2-f9fe-4911-a55b-826a4ce03283'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:57.185684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c8b034f2-f9fe-4911-a55b-826a4ce03283 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] The token cache is being invalidate because project 8294a789da974832918643928932b7a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:57.186352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c8b034f2-f9fe-4911-a55b-826a4ce03283 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:57.188016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c8b034f2-f9fe-4911-a55b-826a4ce03283 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:57.194501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 927/1873] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:56 2022] DELETE /identity/v3/projects/8294a789da974832918643928932b7a1 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:08:57.605278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ef653e-ca14-42de-9c75-2b449dd21e55 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:57.611778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c5ef653e-ca14-42de-9c75-2b449dd21e55 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:57.780893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ef653e-ca14-42de-9c75-2b449dd21e55 tempest-VolumeQuotaClassesTest-1735038064 tempest-VolumeQuotaClassesTest-1735038064-project-admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:57.894638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:57.927742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:57.930072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:57.930393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:57.930850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] PATH_INFO: `/v3/users/e8dd5929d2844bd9bb1e1d52a249d0f4` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:57.933166 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] RBAC: Authorizing `identity:delete_user(user_id=e8dd5929d2844bd9bb1e1d52a249d0f4)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:57.942247 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:58.110910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e8dd5929d2844bd9bb1e1d52a249d0f4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:58.112481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e8dd5929d2844bd9bb1e1d52a249d0f4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:58.141294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e8dd5929d2844bd9bb1e1d52a249d0f4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:58.184996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e8dd5929d2844bd9bb1e1d52a249d0f4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:58.197231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e8dd5929d2844bd9bb1e1d52a249d0f4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:58.207562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 947/1874] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:57 2022] DELETE /identity/v3/users/e8dd5929d2844bd9bb1e1d52a249d0f4 => generated 0 bytes in 604 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:58.214495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6c535b-5f3e-4a1d-a4f0-5754cee40484 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:58.215528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6c535b-5f3e-4a1d-a4f0-5754cee40484 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:58.215528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6c535b-5f3e-4a1d-a4f0-5754cee40484 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:58.237697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-8e6c535b-5f3e-4a1d-a4f0-5754cee40484 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:58.238941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8e6c535b-5f3e-4a1d-a4f0-5754cee40484 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:58.317691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 928/1875] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:08:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:58.327489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:58.328987 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:58.409771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec542856-dd39-4744-a3d5-5d59bb82cb9d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:58.414220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbe42a8-a920-4f73-b913-375f405d1a1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:58.415047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe42a8-a920-4f73-b913-375f405d1a1b service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:58.416342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe42a8-a920-4f73-b913-375f405d1a1b service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:58.416342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe42a8-a920-4f73-b913-375f405d1a1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:58.435519 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbe42a8-a920-4f73-b913-375f405d1a1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:58.438052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbe42a8-a920-4f73-b913-375f405d1a1b service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:58.461179 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 948/1876] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:08:58.538134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa14bab-a575-48c4-aa19-7b124f46459e None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:58.538650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa14bab-a575-48c4-aa19-7b124f46459e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:58.539002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa14bab-a575-48c4-aa19-7b124f46459e None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:58.540900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 929/1877] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:08:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:08:58.561076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6a06ef-06da-48dd-8ab1-bbfac4c54956 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:58.561076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6a06ef-06da-48dd-8ab1-bbfac4c54956 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:58.561251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6a06ef-06da-48dd-8ab1-bbfac4c54956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:58.601240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-8c6a06ef-06da-48dd-8ab1-bbfac4c54956 None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:08:58.602896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8c6a06ef-06da-48dd-8ab1-bbfac4c54956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:58.718064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 949/1878] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:08:58 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:08:58.735903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa14bab-a575-48c4-aa19-7b124f46459e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:58.737927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ffa14bab-a575-48c4-aa19-7b124f46459e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:58.754096 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:58.769474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa14bab-a575-48c4-aa19-7b124f46459e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:58.783903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32317eed-4af4-4c8b-bedc-37e8894bb99f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:58.783903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32317eed-4af4-4c8b-bedc-37e8894bb99f service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:58.784346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32317eed-4af4-4c8b-bedc-37e8894bb99f service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:58.784622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32317eed-4af4-4c8b-bedc-37e8894bb99f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:58.808108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32317eed-4af4-4c8b-bedc-37e8894bb99f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:58.810584 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32317eed-4af4-4c8b-bedc-37e8894bb99f service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:58.827916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 930/1879] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:08:58 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:08:58.906682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6a06ef-06da-48dd-8ab1-bbfac4c54956 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:58.909001 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8c6a06ef-06da-48dd-8ab1-bbfac4c54956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:58.936369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6a06ef-06da-48dd-8ab1-bbfac4c54956 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:58.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:58.946967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:58.947506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:58.947941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] PATH_INFO: `/v3/projects/926a1fb02a7c435b9e1ec997271536e2` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:58.949492 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=926a1fb02a7c435b9e1ec997271536e2)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:58.952408 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:58.955803 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:08:59.098187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '926a1fb02a7c435b9e1ec997271536e2', 'request_id': 'req-39288769-c942-4bac-8f9f-85f9527aa7a3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:59.109140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '926a1fb02a7c435b9e1ec997271536e2', 'request_id': 'req-39288769-c942-4bac-8f9f-85f9527aa7a3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:59.120071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '926a1fb02a7c435b9e1ec997271536e2', 'request_id': 'req-39288769-c942-4bac-8f9f-85f9527aa7a3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:59.136847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] The token cache is being invalidate because project 926a1fb02a7c435b9e1ec997271536e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:08:59.137271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:59.138386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:59.140399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 950/1880] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:58 2022] DELETE /identity/v3/projects/926a1fb02a7c435b9e1ec997271536e2 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:59.151609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32317eed-4af4-4c8b-bedc-37e8894bb99f service nova] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:59.179430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32317eed-4af4-4c8b-bedc-37e8894bb99f service nova] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:59.316773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9417131-90d9-4157-8582-8f17f44ac2c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:59.317489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9417131-90d9-4157-8582-8f17f44ac2c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:59.317725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9417131-90d9-4157-8582-8f17f44ac2c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:59.318020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9417131-90d9-4157-8582-8f17f44ac2c4 admin admin] PATH_INFO: `/v3/users/d8405dc1f3024f97849e871164cb07ad` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:59.320700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9417131-90d9-4157-8582-8f17f44ac2c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8405dc1f3024f97849e871164cb07ad)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:59.323640 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9417131-90d9-4157-8582-8f17f44ac2c4 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:08:59.404907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a9417131-90d9-4157-8582-8f17f44ac2c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8405dc1f3024f97849e871164cb07ad'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:59.415936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a9417131-90d9-4157-8582-8f17f44ac2c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8405dc1f3024f97849e871164cb07ad'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:59.416754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a9417131-90d9-4157-8582-8f17f44ac2c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8405dc1f3024f97849e871164cb07ad'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:59.435020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a9417131-90d9-4157-8582-8f17f44ac2c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8405dc1f3024f97849e871164cb07ad'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:59.435791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a9417131-90d9-4157-8582-8f17f44ac2c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8405dc1f3024f97849e871164cb07ad'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:08:59.458792 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 931/1881] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:08:59 2022] DELETE /identity/v3/users/d8405dc1f3024f97849e871164cb07ad => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:08:59.812258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:08:59.816211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:08:59.852743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39288769-c942-4bac-8f9f-85f9527aa7a3 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:08:59.937802 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d46866-c114-4109-8e79-1e7287fd6c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:08:59.938860 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d46866-c114-4109-8e79-1e7287fd6c1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:08:59.939053 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d46866-c114-4109-8e79-1e7287fd6c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:08:59.939451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d46866-c114-4109-8e79-1e7287fd6c1c admin admin] PATH_INFO: `/v3/projects/78a43db51b7e4c4183fee24dbc6ddaa6` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:08:59.941110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d46866-c114-4109-8e79-1e7287fd6c1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=78a43db51b7e4c4183fee24dbc6ddaa6)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:08:59.943327 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d46866-c114-4109-8e79-1e7287fd6c1c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:00.031433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87d46866-c114-4109-8e79-1e7287fd6c1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78a43db51b7e4c4183fee24dbc6ddaa6', 'request_id': 'req-87d46866-c114-4109-8e79-1e7287fd6c1c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:00.040065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87d46866-c114-4109-8e79-1e7287fd6c1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78a43db51b7e4c4183fee24dbc6ddaa6', 'request_id': 'req-87d46866-c114-4109-8e79-1e7287fd6c1c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:00.055009 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87d46866-c114-4109-8e79-1e7287fd6c1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78a43db51b7e4c4183fee24dbc6ddaa6', 'request_id': 'req-87d46866-c114-4109-8e79-1e7287fd6c1c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:00.073949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87d46866-c114-4109-8e79-1e7287fd6c1c admin admin] The token cache is being invalidate because project 78a43db51b7e4c4183fee24dbc6ddaa6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:00.074294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87d46866-c114-4109-8e79-1e7287fd6c1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:00.075001 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87d46866-c114-4109-8e79-1e7287fd6c1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:00.080685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 951/1882] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:08:59 2022] DELETE /identity/v3/projects/78a43db51b7e4c4183fee24dbc6ddaa6 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:00.084470 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86112426-012f-4198-b3f5-4bb91a57a4d3 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:00.084914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86112426-012f-4198-b3f5-4bb91a57a4d3 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:00.085198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86112426-012f-4198-b3f5-4bb91a57a4d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:00.137191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-86112426-012f-4198-b3f5-4bb91a57a4d3 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:00.138840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-86112426-012f-4198-b3f5-4bb91a57a4d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:00.233312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 932/1883] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:00.245926 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfff1cf1-7a7d-45b0-9581-930f6df277df None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:00.246477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfff1cf1-7a7d-45b0-9581-930f6df277df None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:00.247079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfff1cf1-7a7d-45b0-9581-930f6df277df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:00.305392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-dfff1cf1-7a7d-45b0-9581-930f6df277df None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:00.308615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-dfff1cf1-7a7d-45b0-9581-930f6df277df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:00.327035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 952/1884] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:00.335052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86112426-012f-4198-b3f5-4bb91a57a4d3 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:00.360588 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86112426-012f-4198-b3f5-4bb91a57a4d3 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:00.394557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e07cbae-bdbf-4e5c-a87c-3a6e982c666a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:00.395742 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e07cbae-bdbf-4e5c-a87c-3a6e982c666a admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:00.396471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e07cbae-bdbf-4e5c-a87c-3a6e982c666a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:00.397199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e07cbae-bdbf-4e5c-a87c-3a6e982c666a admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:00.399250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e07cbae-bdbf-4e5c-a87c-3a6e982c666a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:00.399703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e07cbae-bdbf-4e5c-a87c-3a6e982c666a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:00.402510 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e07cbae-bdbf-4e5c-a87c-3a6e982c666a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:00.408204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:00.429261 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 933/1885] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:09:00.437892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfff1cf1-7a7d-45b0-9581-930f6df277df None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:00.459866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfff1cf1-7a7d-45b0-9581-930f6df277df None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:00.477540 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b67a62a-3229-4af6-9edd-bf0cfb140514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:00.479048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b67a62a-3229-4af6-9edd-bf0cfb140514 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:00.479788 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b67a62a-3229-4af6-9edd-bf0cfb140514 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:00.480439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b67a62a-3229-4af6-9edd-bf0cfb140514 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:00.481930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b67a62a-3229-4af6-9edd-bf0cfb140514 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:00.487675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b67a62a-3229-4af6-9edd-bf0cfb140514 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:00.568013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:00.625334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 953/1886] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:00 2022] POST /identity/v3/projects => generated 387 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:00.628643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e07cbae-bdbf-4e5c-a87c-3a6e982c666a admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:00.660969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e07cbae-bdbf-4e5c-a87c-3a6e982c666a admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:00.823299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b67a62a-3229-4af6-9edd-bf0cfb140514 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:00.825599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0b67a62a-3229-4af6-9edd-bf0cfb140514 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:00.887461 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30566c7b-7ea5-41a1-8d13-2d70563acd86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:00.888266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30566c7b-7ea5-41a1-8d13-2d70563acd86 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:00.889757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30566c7b-7ea5-41a1-8d13-2d70563acd86 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:00.891058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30566c7b-7ea5-41a1-8d13-2d70563acd86 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:00.894044 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30566c7b-7ea5-41a1-8d13-2d70563acd86 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:00.896263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30566c7b-7ea5-41a1-8d13-2d70563acd86 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:00.979738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 934/1887] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:00 2022] POST /identity/v3/users => generated 294 bytes in 352 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:00.984847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30566c7b-7ea5-41a1-8d13-2d70563acd86 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:01.014361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30566c7b-7ea5-41a1-8d13-2d70563acd86 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:01.029410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4d6475-df4c-447f-93c3-7d5849a61282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:01.030533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4d6475-df4c-447f-93c3-7d5849a61282 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:01.031879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4d6475-df4c-447f-93c3-7d5849a61282 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:01.032508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4d6475-df4c-447f-93c3-7d5849a61282 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:01.034181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4d6475-df4c-447f-93c3-7d5849a61282 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:01.042365 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4d6475-df4c-447f-93c3-7d5849a61282 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:01.080819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 935/1888] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:00 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:01.082936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4d6475-df4c-447f-93c3-7d5849a61282 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:01.105969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4d6475-df4c-447f-93c3-7d5849a61282 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:01.120519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee7637d-63b9-4e7a-bd5e-cb5954e213ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:01.120716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b67a62a-3229-4af6-9edd-bf0cfb140514 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:01.122666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee7637d-63b9-4e7a-bd5e-cb5954e213ca admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:01.140660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee7637d-63b9-4e7a-bd5e-cb5954e213ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:01.143075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee7637d-63b9-4e7a-bd5e-cb5954e213ca admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:01.146217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36d6b0f-5f08-4b0b-a9cf-3099ea48d567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:01.146473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee7637d-63b9-4e7a-bd5e-cb5954e213ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:01.147966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36d6b0f-5f08-4b0b-a9cf-3099ea48d567 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:01.148729 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36d6b0f-5f08-4b0b-a9cf-3099ea48d567 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:01.149488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36d6b0f-5f08-4b0b-a9cf-3099ea48d567 admin admin] PATH_INFO: `/v3/users/ecc4c980fe9c43ad9998e93ab4b0c32f` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:01.153336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36d6b0f-5f08-4b0b-a9cf-3099ea48d567 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ecc4c980fe9c43ad9998e93ab4b0c32f)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:01.157358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee7637d-63b9-4e7a-bd5e-cb5954e213ca admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:01.159939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36d6b0f-5f08-4b0b-a9cf-3099ea48d567 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:01.198487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 936/1889] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:01 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:01.210471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee7637d-63b9-4e7a-bd5e-cb5954e213ca admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:01.256825 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee7637d-63b9-4e7a-bd5e-cb5954e213ca admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:01.274149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ef1e14-7e33-49b3-a23e-5e9a654d1098 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:01.280444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ef1e14-7e33-49b3-a23e-5e9a654d1098 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:01.282132 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ef1e14-7e33-49b3-a23e-5e9a654d1098 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:01.283667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ef1e14-7e33-49b3-a23e-5e9a654d1098 admin admin] PATH_INFO: `/v3/projects/9afc8112c0d54bdf987ef0c44d572471/users/44e3542b1dd5485e89c1804bc6467462/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:01.345603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-a36d6b0f-5f08-4b0b-a9cf-3099ea48d567 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ecc4c980fe9c43ad9998e93ab4b0c32f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:01.345911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ef1e14-7e33-49b3-a23e-5e9a654d1098 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9afc8112c0d54bdf987ef0c44d572471, user_id=44e3542b1dd5485e89c1804bc6467462, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:01.348042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-a36d6b0f-5f08-4b0b-a9cf-3099ea48d567 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ecc4c980fe9c43ad9998e93ab4b0c32f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:01.351915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ef1e14-7e33-49b3-a23e-5e9a654d1098 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:01.363667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-a36d6b0f-5f08-4b0b-a9cf-3099ea48d567 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ecc4c980fe9c43ad9998e93ab4b0c32f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:01.381841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 937/1890] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:01 2022] PUT /identity/v3/projects/9afc8112c0d54bdf987ef0c44d572471/users/44e3542b1dd5485e89c1804bc6467462/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:01.394901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef293e56-20b9-4a04-b217-4c52f9627a7b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:01.395544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef293e56-20b9-4a04-b217-4c52f9627a7b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:01.396044 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef293e56-20b9-4a04-b217-4c52f9627a7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:01.409791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-a36d6b0f-5f08-4b0b-a9cf-3099ea48d567 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ecc4c980fe9c43ad9998e93ab4b0c32f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:01.443200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-a36d6b0f-5f08-4b0b-a9cf-3099ea48d567 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ecc4c980fe9c43ad9998e93ab4b0c32f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:01.456406 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 954/1891] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:00 2022] DELETE /identity/v3/users/ecc4c980fe9c43ad9998e93ab4b0c32f => generated 0 bytes in 634 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:01.456791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ef293e56-20b9-4a04-b217-4c52f9627a7b None None] MFA Rules not processed for user `44e3542b1dd5485e89c1804bc6467462`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:01.459032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ef293e56-20b9-4a04-b217-4c52f9627a7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:01.462681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4de760f-3b1e-4977-8d02-f200b6679ad9 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:01.464149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4de760f-3b1e-4977-8d02-f200b6679ad9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:01.464701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4de760f-3b1e-4977-8d02-f200b6679ad9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:01.494898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-c4de760f-3b1e-4977-8d02-f200b6679ad9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:01.496465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c4de760f-3b1e-4977-8d02-f200b6679ad9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:01.639266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 955/1892] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:01.650817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4de760f-3b1e-4977-8d02-f200b6679ad9 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:01.652397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c4de760f-3b1e-4977-8d02-f200b6679ad9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:01.680323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 938/1893] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:01 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:01.685267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef293e56-20b9-4a04-b217-4c52f9627a7b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:01.687915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ef293e56-20b9-4a04-b217-4c52f9627a7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:01.693183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:01.711610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef293e56-20b9-4a04-b217-4c52f9627a7b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:01.716856 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69afa4e3-6cfb-4299-b4c7-304afde8e9a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:01.717732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69afa4e3-6cfb-4299-b4c7-304afde8e9a5 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:01.718272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69afa4e3-6cfb-4299-b4c7-304afde8e9a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:01.718857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69afa4e3-6cfb-4299-b4c7-304afde8e9a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:01.719819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69afa4e3-6cfb-4299-b4c7-304afde8e9a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:01.722042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69afa4e3-6cfb-4299-b4c7-304afde8e9a5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:01.767221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 939/1894] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:01 2022] POST /identity/v3/projects => generated 387 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:09:01.772866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69afa4e3-6cfb-4299-b4c7-304afde8e9a5 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:01.786270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4de760f-3b1e-4977-8d02-f200b6679ad9 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:01.786960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:01.874437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69afa4e3-6cfb-4299-b4c7-304afde8e9a5 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:01.880374 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f371548-c2a6-45cd-ac9d-1c3a9aa00586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:01.881019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f371548-c2a6-45cd-ac9d-1c3a9aa00586 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:01.881263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e589eb4-9548-4daf-aa3b-ae8c3a2585c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:01.881393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f371548-c2a6-45cd-ac9d-1c3a9aa00586 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:01.881566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f371548-c2a6-45cd-ac9d-1c3a9aa00586 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:01.882244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f371548-c2a6-45cd-ac9d-1c3a9aa00586 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:01.882352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e589eb4-9548-4daf-aa3b-ae8c3a2585c2 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:01.882794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e589eb4-9548-4daf-aa3b-ae8c3a2585c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:01.883300 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e589eb4-9548-4daf-aa3b-ae8c3a2585c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:01.884417 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f371548-c2a6-45cd-ac9d-1c3a9aa00586 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:01.902254 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e589eb4-9548-4daf-aa3b-ae8c3a2585c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:01.904947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e589eb4-9548-4daf-aa3b-ae8c3a2585c2 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:01.910737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 940/1895] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:01 2022] POST /identity/v3/users => generated 294 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:01.915409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f371548-c2a6-45cd-ac9d-1c3a9aa00586 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:01.921012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 956/1896] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:01.930419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f371548-c2a6-45cd-ac9d-1c3a9aa00586 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:01.940935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791978e2-817c-40e2-aa07-76b673c71422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:01.942012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791978e2-817c-40e2-aa07-76b673c71422 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:01.942505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791978e2-817c-40e2-aa07-76b673c71422 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:01.943002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791978e2-817c-40e2-aa07-76b673c71422 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:01.944366 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791978e2-817c-40e2-aa07-76b673c71422 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:01.947467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791978e2-817c-40e2-aa07-76b673c71422 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:01.970276 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 941/1897] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:01 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:01.972047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e589eb4-9548-4daf-aa3b-ae8c3a2585c2 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:01.984389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e589eb4-9548-4daf-aa3b-ae8c3a2585c2 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:01.992199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06bb27c-64fb-4406-8e20-b5f098072891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:01.992853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06bb27c-64fb-4406-8e20-b5f098072891 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:01.994047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06bb27c-64fb-4406-8e20-b5f098072891 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:01.994376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06bb27c-64fb-4406-8e20-b5f098072891 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:01.995126 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06bb27c-64fb-4406-8e20-b5f098072891 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:01.997919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06bb27c-64fb-4406-8e20-b5f098072891 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:02.018158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 957/1898] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:01 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:02.020027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791978e2-817c-40e2-aa07-76b673c71422 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:02.052268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791978e2-817c-40e2-aa07-76b673c71422 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:02.061938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45c19b8-cbc9-41c1-acbf-30c930f161f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:02.063121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45c19b8-cbc9-41c1-acbf-30c930f161f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:02.064565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45c19b8-cbc9-41c1-acbf-30c930f161f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:02.065418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45c19b8-cbc9-41c1-acbf-30c930f161f9 admin admin] PATH_INFO: `/v3/projects/8d82f770c9534c14b4cec601d5dd68b9/users/4318271b271749aeaaa74adee2651b8c/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:02.113078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45c19b8-cbc9-41c1-acbf-30c930f161f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d82f770c9534c14b4cec601d5dd68b9, user_id=4318271b271749aeaaa74adee2651b8c, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:02.129224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45c19b8-cbc9-41c1-acbf-30c930f161f9 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:02.159057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 942/1899] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:02 2022] PUT /identity/v3/projects/8d82f770c9534c14b4cec601d5dd68b9/users/4318271b271749aeaaa74adee2651b8c/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:02.171324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210130c2-3595-4876-af1a-3e4c17048c79 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:02.171324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210130c2-3595-4876-af1a-3e4c17048c79 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:02.171324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210130c2-3595-4876-af1a-3e4c17048c79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:02.242905 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-210130c2-3595-4876-af1a-3e4c17048c79 None None] MFA Rules not processed for user `4318271b271749aeaaa74adee2651b8c`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:02.245768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-210130c2-3595-4876-af1a-3e4c17048c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:02.374596 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 958/1900] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:02 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:02.377652 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45c19b8-cbc9-41c1-acbf-30c930f161f9 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:02.457170 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210130c2-3595-4876-af1a-3e4c17048c79 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:02.459885 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-210130c2-3595-4876-af1a-3e4c17048c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:02.533074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45c19b8-cbc9-41c1-acbf-30c930f161f9 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:02.542310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3729db13-6037-4d06-b29d-584ef462d4a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:02.543521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729db13-6037-4d06-b29d-584ef462d4a6 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:02.543756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729db13-6037-4d06-b29d-584ef462d4a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:02.544034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729db13-6037-4d06-b29d-584ef462d4a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:02.544774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729db13-6037-4d06-b29d-584ef462d4a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:02.549514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729db13-6037-4d06-b29d-584ef462d4a6 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:02.574199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:02.607297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210130c2-3595-4876-af1a-3e4c17048c79 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:02.614878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 943/1901] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:02 2022] POST /identity/v3/projects => generated 383 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:02.616485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3729db13-6037-4d06-b29d-584ef462d4a6 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:02.713948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:02.715440 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:02.715801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:02.716152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] PATH_INFO: `/v3/projects/3c9901d2ab45408f9662f17986097fe9` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:02.717813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c9901d2ab45408f9662f17986097fe9)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:02.720274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:02.747273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:02.792020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3729db13-6037-4d06-b29d-584ef462d4a6 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:02.817964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f6b49a-9a3c-422c-84a2-1e8967b0dc24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:02.820242 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f6b49a-9a3c-422c-84a2-1e8967b0dc24 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:02.820645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f6b49a-9a3c-422c-84a2-1e8967b0dc24 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:02.820962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c9901d2ab45408f9662f17986097fe9', 'request_id': 'req-da496ed2-6291-4883-84b6-6acf68cfa8bf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:02.821142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f6b49a-9a3c-422c-84a2-1e8967b0dc24 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:02.821848 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f6b49a-9a3c-422c-84a2-1e8967b0dc24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:02.825320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f6b49a-9a3c-422c-84a2-1e8967b0dc24 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:02.829863 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c9901d2ab45408f9662f17986097fe9', 'request_id': 'req-da496ed2-6291-4883-84b6-6acf68cfa8bf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:02.842681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c9901d2ab45408f9662f17986097fe9', 'request_id': 'req-da496ed2-6291-4883-84b6-6acf68cfa8bf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:02.852642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 944/1902] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:02 2022] POST /identity/v3/users => generated 298 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:02.858330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f6b49a-9a3c-422c-84a2-1e8967b0dc24 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:02.866278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] The token cache is being invalidate because project 3c9901d2ab45408f9662f17986097fe9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:02.866806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:02.867970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:02.870753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 959/1903] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:02 2022] DELETE /identity/v3/projects/3c9901d2ab45408f9662f17986097fe9 => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:02.871549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f6b49a-9a3c-422c-84a2-1e8967b0dc24 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:02.876376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:02.879078 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:02.943889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efafc786-4423-4f20-b729-1ed261e09cee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:02.944767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efafc786-4423-4f20-b729-1ed261e09cee admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:02.945267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efafc786-4423-4f20-b729-1ed261e09cee admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:02.945912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efafc786-4423-4f20-b729-1ed261e09cee admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:02.946978 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efafc786-4423-4f20-b729-1ed261e09cee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:02.950410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efafc786-4423-4f20-b729-1ed261e09cee admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:02.972474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 945/1904] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:02 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:09:02.977406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efafc786-4423-4f20-b729-1ed261e09cee admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:02.979599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-efafc786-4423-4f20-b729-1ed261e09cee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:02.996715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efafc786-4423-4f20-b729-1ed261e09cee admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:03.004267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa24c4ba-75fe-41d0-8d6c-97ab6c384d49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:03.004881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa24c4ba-75fe-41d0-8d6c-97ab6c384d49 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:03.005360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa24c4ba-75fe-41d0-8d6c-97ab6c384d49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:03.005638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa24c4ba-75fe-41d0-8d6c-97ab6c384d49 admin admin] PATH_INFO: `/v3/projects/ade3f2d21b714fdf861390c0c6069758/users/375882268c5d4366b8411c62e2200aa3/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:03.011458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da496ed2-6291-4883-84b6-6acf68cfa8bf admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:03.020041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b6ac5c-5e08-48a0-9671-f7b7dd6ea309 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:03.021166 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b6ac5c-5e08-48a0-9671-f7b7dd6ea309 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:03.021672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b6ac5c-5e08-48a0-9671-f7b7dd6ea309 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:03.022159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b6ac5c-5e08-48a0-9671-f7b7dd6ea309 admin admin] PATH_INFO: `/v3/users/cce60d9fb54144b59e7bccec438a36a4` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:03.024666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b6ac5c-5e08-48a0-9671-f7b7dd6ea309 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cce60d9fb54144b59e7bccec438a36a4)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:03.028756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b6ac5c-5e08-48a0-9671-f7b7dd6ea309 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:03.029057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa24c4ba-75fe-41d0-8d6c-97ab6c384d49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ade3f2d21b714fdf861390c0c6069758, user_id=375882268c5d4366b8411c62e2200aa3, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:03.032957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa24c4ba-75fe-41d0-8d6c-97ab6c384d49 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:03.057717 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 946/1905] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:02 2022] PUT /identity/v3/projects/ade3f2d21b714fdf861390c0c6069758/users/375882268c5d4366b8411c62e2200aa3/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:03.058721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa24c4ba-75fe-41d0-8d6c-97ab6c384d49 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:03.092144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:03.165326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-57b6ac5c-5e08-48a0-9671-f7b7dd6ea309 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cce60d9fb54144b59e7bccec438a36a4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:03.168703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-57b6ac5c-5e08-48a0-9671-f7b7dd6ea309 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cce60d9fb54144b59e7bccec438a36a4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:03.200427 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-57b6ac5c-5e08-48a0-9671-f7b7dd6ea309 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cce60d9fb54144b59e7bccec438a36a4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:03.207432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa24c4ba-75fe-41d0-8d6c-97ab6c384d49 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:03.230249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517cc8c8-f657-4a73-8b2a-8f22de7ed3e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:03.230249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517cc8c8-f657-4a73-8b2a-8f22de7ed3e8 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:03.230249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517cc8c8-f657-4a73-8b2a-8f22de7ed3e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:03.230249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517cc8c8-f657-4a73-8b2a-8f22de7ed3e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:03.230786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517cc8c8-f657-4a73-8b2a-8f22de7ed3e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:03.233469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517cc8c8-f657-4a73-8b2a-8f22de7ed3e8 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:03.241209 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-57b6ac5c-5e08-48a0-9671-f7b7dd6ea309 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cce60d9fb54144b59e7bccec438a36a4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:03.251615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 947/1906] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:03 2022] GET /identity/v3/roles => generated 1420 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:03.255995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517cc8c8-f657-4a73-8b2a-8f22de7ed3e8 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:03.260614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-57b6ac5c-5e08-48a0-9671-f7b7dd6ea309 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cce60d9fb54144b59e7bccec438a36a4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:03.273603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517cc8c8-f657-4a73-8b2a-8f22de7ed3e8 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:03.276378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 960/1907] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:02 2022] DELETE /identity/v3/users/cce60d9fb54144b59e7bccec438a36a4 => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:03.282824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252b38e3-65fb-4ea3-9edc-0e7dcef6a3d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:03.283569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252b38e3-65fb-4ea3-9edc-0e7dcef6a3d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:03.283856 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252b38e3-65fb-4ea3-9edc-0e7dcef6a3d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:03.284111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252b38e3-65fb-4ea3-9edc-0e7dcef6a3d2 admin admin] PATH_INFO: `/v3/projects/ade3f2d21b714fdf861390c0c6069758/users/375882268c5d4366b8411c62e2200aa3/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:03.286251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252b38e3-65fb-4ea3-9edc-0e7dcef6a3d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ade3f2d21b714fdf861390c0c6069758, user_id=375882268c5d4366b8411c62e2200aa3, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:03.289053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252b38e3-65fb-4ea3-9edc-0e7dcef6a3d2 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:03.302579 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 948/1908] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:03 2022] PUT /identity/v3/projects/ade3f2d21b714fdf861390c0c6069758/users/375882268c5d4366b8411c62e2200aa3/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:03.320888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3fa24c-9b0a-4e9e-9286-0a96cfbc105d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:03.321340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3fa24c-9b0a-4e9e-9286-0a96cfbc105d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:03.321684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3fa24c-9b0a-4e9e-9286-0a96cfbc105d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:03.365932 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-dc3fa24c-9b0a-4e9e-9286-0a96cfbc105d None None] MFA Rules not processed for user `375882268c5d4366b8411c62e2200aa3`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:03.367461 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-dc3fa24c-9b0a-4e9e-9286-0a96cfbc105d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:03.479750 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 961/1909] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:03 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:03.491744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252b38e3-65fb-4ea3-9edc-0e7dcef6a3d2 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:03.493940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-252b38e3-65fb-4ea3-9edc-0e7dcef6a3d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:03.558483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:03.588464 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252b38e3-65fb-4ea3-9edc-0e7dcef6a3d2 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:03.592307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f43917-a82e-451f-b5cc-b5afd85f3b8e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:03.592956 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f43917-a82e-451f-b5cc-b5afd85f3b8e service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:03.593240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f43917-a82e-451f-b5cc-b5afd85f3b8e service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:03.593507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f43917-a82e-451f-b5cc-b5afd85f3b8e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:03.604274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f43917-a82e-451f-b5cc-b5afd85f3b8e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:03.605743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f43917-a82e-451f-b5cc-b5afd85f3b8e service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:03.617831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 949/1910] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:03 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:03.712636 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3fa24c-9b0a-4e9e-9286-0a96cfbc105d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:03.716290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-dc3fa24c-9b0a-4e9e-9286-0a96cfbc105d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:03.837046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3fa24c-9b0a-4e9e-9286-0a96cfbc105d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:03.854315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:03.857125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:03.857125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:03.857125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] PATH_INFO: `/v3/projects/444d903f04c540c5ab06974e609b4923` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:03.858918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=444d903f04c540c5ab06974e609b4923)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:03.862591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:03.923318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:03.959888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '444d903f04c540c5ab06974e609b4923', 'request_id': 'req-4d913035-4d13-44e7-95c6-2dbc03aa9d57'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:03.959888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '444d903f04c540c5ab06974e609b4923', 'request_id': 'req-4d913035-4d13-44e7-95c6-2dbc03aa9d57'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:03.969549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '444d903f04c540c5ab06974e609b4923', 'request_id': 'req-4d913035-4d13-44e7-95c6-2dbc03aa9d57'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:03.987741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] The token cache is being invalidate because project 444d903f04c540c5ab06974e609b4923 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:03.988340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:03.989332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:03.991555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 962/1911] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:03 2022] DELETE /identity/v3/projects/444d903f04c540c5ab06974e609b4923 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:03.996176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f43917-a82e-451f-b5cc-b5afd85f3b8e service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:04.015017 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f43917-a82e-451f-b5cc-b5afd85f3b8e service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:04.089220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:04.090026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:04.090366 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:04.090711 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] PATH_INFO: `/v3/users/0b030fbbccd94f71ba645e1acb626e73` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:04.092077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b030fbbccd94f71ba645e1acb626e73)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:04.094096 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:04.163612 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b030fbbccd94f71ba645e1acb626e73'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:04.174889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b030fbbccd94f71ba645e1acb626e73'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:04.175881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b030fbbccd94f71ba645e1acb626e73'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:04.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b030fbbccd94f71ba645e1acb626e73'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:04.198312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b030fbbccd94f71ba645e1acb626e73'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:04.215831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 950/1912] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:03 2022] DELETE /identity/v3/users/0b030fbbccd94f71ba645e1acb626e73 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:04.467244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:04.471471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:04.608630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d913035-4d13-44e7-95c6-2dbc03aa9d57 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:04.732033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e70231-bd75-4063-bbe2-318aa00534a3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:04.733257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e70231-bd75-4063-bbe2-318aa00534a3 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:04.733755 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e70231-bd75-4063-bbe2-318aa00534a3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:04.734315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e70231-bd75-4063-bbe2-318aa00534a3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:04.738084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c7e70231-bd75-4063-bbe2-318aa00534a3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:04.856557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:04.858930 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:04.859422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e70231-bd75-4063-bbe2-318aa00534a3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:04.862451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e70231-bd75-4063-bbe2-318aa00534a3 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:04.907956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 963/1913] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:04 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:04.938831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:04.968381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e73342-398c-4873-a09c-55b758989658 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:04.977880 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:04.978680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:04.979034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:04.979409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a admin admin] PATH_INFO: `/v3/projects/d7f45101a3d24ee8a180b15a3298b20b` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:04.980840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a admin admin] RBAC: Authorizing `identity:delete_project(project_id=d7f45101a3d24ee8a180b15a3298b20b)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:04.983188 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:05.068386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7f45101a3d24ee8a180b15a3298b20b', 'request_id': 'req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:05.084107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7f45101a3d24ee8a180b15a3298b20b', 'request_id': 'req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:05.105363 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7f45101a3d24ee8a180b15a3298b20b', 'request_id': 'req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:05.128233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a admin admin] The token cache is being invalidate because project d7f45101a3d24ee8a180b15a3298b20b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:05.128655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:05.129788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0bdfb776-aa66-4e1e-8d5f-db3adc95135a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:05.133959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 951/1914] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:04 2022] DELETE /identity/v3/projects/d7f45101a3d24ee8a180b15a3298b20b => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:05.145442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee852e3-1ae5-44b2-af85-be3e21e9f401 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:05.146044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee852e3-1ae5-44b2-af85-be3e21e9f401 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:05.148882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee852e3-1ae5-44b2-af85-be3e21e9f401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:05.178848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-bee852e3-1ae5-44b2-af85-be3e21e9f401 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:05.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-bee852e3-1ae5-44b2-af85-be3e21e9f401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:05.217427 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:05.269279 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 964/1915] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:05.280002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3d9f3d-1fb4-47f8-92bb-6d88574e1780 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:05.280523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3d9f3d-1fb4-47f8-92bb-6d88574e1780 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:05.281025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3d9f3d-1fb4-47f8-92bb-6d88574e1780 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:05.321644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6b3d9f3d-1fb4-47f8-92bb-6d88574e1780 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:05.324090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6b3d9f3d-1fb4-47f8-92bb-6d88574e1780 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:05.340240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 952/1916] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:05.347743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee852e3-1ae5-44b2-af85-be3e21e9f401 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:05.367962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee852e3-1ae5-44b2-af85-be3e21e9f401 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:05.377067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20442570-66e1-428f-b953-b8d4851b795f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:05.378162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20442570-66e1-428f-b953-b8d4851b795f admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:05.378641 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20442570-66e1-428f-b953-b8d4851b795f admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:05.379133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20442570-66e1-428f-b953-b8d4851b795f admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:05.381351 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20442570-66e1-428f-b953-b8d4851b795f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:05.382039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20442570-66e1-428f-b953-b8d4851b795f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:05.385520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20442570-66e1-428f-b953-b8d4851b795f admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:05.415424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 965/1917] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:09:05.422176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3d9f3d-1fb4-47f8-92bb-6d88574e1780 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:05.443415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3d9f3d-1fb4-47f8-92bb-6d88574e1780 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:05.451638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca7c6dc-33a7-4d66-9a49-0a09c426f505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:05.452661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca7c6dc-33a7-4d66-9a49-0a09c426f505 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:05.453124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca7c6dc-33a7-4d66-9a49-0a09c426f505 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:05.453567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca7c6dc-33a7-4d66-9a49-0a09c426f505 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:05.455473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca7c6dc-33a7-4d66-9a49-0a09c426f505 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:05.458874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca7c6dc-33a7-4d66-9a49-0a09c426f505 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:05.501269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 953/1918] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:05 2022] POST /identity/v3/projects => generated 373 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:05.507115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20442570-66e1-428f-b953-b8d4851b795f admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:05.520665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20442570-66e1-428f-b953-b8d4851b795f admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:05.611395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d3257e-2d45-420c-afae-77bd9d25f637 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:05.612020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d3257e-2d45-420c-afae-77bd9d25f637 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:05.612299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d3257e-2d45-420c-afae-77bd9d25f637 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:05.612593 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d3257e-2d45-420c-afae-77bd9d25f637 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:05.613249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d3257e-2d45-420c-afae-77bd9d25f637 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:05.615242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d3257e-2d45-420c-afae-77bd9d25f637 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:05.638760 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 966/1919] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:05 2022] POST /identity/v3/users => generated 287 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:05.645836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca7c6dc-33a7-4d66-9a49-0a09c426f505 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:05.663997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca7c6dc-33a7-4d66-9a49-0a09c426f505 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:05.672179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89bf1e2-867f-4839-893c-3d9b3a855ad5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:05.674428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89bf1e2-867f-4839-893c-3d9b3a855ad5 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:05.674428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89bf1e2-867f-4839-893c-3d9b3a855ad5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:05.674897 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89bf1e2-867f-4839-893c-3d9b3a855ad5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:05.676013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89bf1e2-867f-4839-893c-3d9b3a855ad5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:05.679141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89bf1e2-867f-4839-893c-3d9b3a855ad5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:05.699169 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 954/1920] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:05 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:05.704306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d3257e-2d45-420c-afae-77bd9d25f637 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:05.717664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d3257e-2d45-420c-afae-77bd9d25f637 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:05.724370 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aaeba16-0d65-479b-97b3-419f0a43545a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:05.725171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aaeba16-0d65-479b-97b3-419f0a43545a admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:05.725569 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aaeba16-0d65-479b-97b3-419f0a43545a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:05.725958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aaeba16-0d65-479b-97b3-419f0a43545a admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:05.726934 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aaeba16-0d65-479b-97b3-419f0a43545a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:05.728997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aaeba16-0d65-479b-97b3-419f0a43545a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:05.742089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 967/1921] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:05 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:05.746517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89bf1e2-867f-4839-893c-3d9b3a855ad5 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:05.758361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89bf1e2-867f-4839-893c-3d9b3a855ad5 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:05.763571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6062d99-a1a5-4c3c-b9d9-8799dc09ed67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:05.765010 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6062d99-a1a5-4c3c-b9d9-8799dc09ed67 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:05.765010 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6062d99-a1a5-4c3c-b9d9-8799dc09ed67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:05.765010 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6062d99-a1a5-4c3c-b9d9-8799dc09ed67 admin admin] PATH_INFO: `/v3/projects/4989caebd6cf4d7da7278b00ef14ef50/users/6b37ea618a944f6591d2089905b39c29/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:05.786800 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6062d99-a1a5-4c3c-b9d9-8799dc09ed67 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4989caebd6cf4d7da7278b00ef14ef50, user_id=6b37ea618a944f6591d2089905b39c29, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:05.789863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6062d99-a1a5-4c3c-b9d9-8799dc09ed67 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:05.807815 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 955/1922] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:05 2022] PUT /identity/v3/projects/4989caebd6cf4d7da7278b00ef14ef50/users/6b37ea618a944f6591d2089905b39c29/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:05.830332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ed783-fab4-4cd2-807f-ca7fb5d81091 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:05.830805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ed783-fab4-4cd2-807f-ca7fb5d81091 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:05.831269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ed783-fab4-4cd2-807f-ca7fb5d81091 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:05.878615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-307ed783-fab4-4cd2-807f-ca7fb5d81091 None None] MFA Rules not processed for user `6b37ea618a944f6591d2089905b39c29`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:05.880624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-307ed783-fab4-4cd2-807f-ca7fb5d81091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:05.997795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 968/1923] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:05 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:06.004354 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6062d99-a1a5-4c3c-b9d9-8799dc09ed67 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:06.081117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6062d99-a1a5-4c3c-b9d9-8799dc09ed67 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:06.086047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fb9fa9-893b-446a-88ed-c365d08e620f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:06.086835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fb9fa9-893b-446a-88ed-c365d08e620f admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:06.087159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fb9fa9-893b-446a-88ed-c365d08e620f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:06.087542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fb9fa9-893b-446a-88ed-c365d08e620f admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:06.088276 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fb9fa9-893b-446a-88ed-c365d08e620f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:06.090283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fb9fa9-893b-446a-88ed-c365d08e620f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:06.130423 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 956/1924] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:06 2022] POST /identity/v3/projects => generated 375 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:09:06.135934 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307ed783-fab4-4cd2-807f-ca7fb5d81091 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:06.150601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307ed783-fab4-4cd2-807f-ca7fb5d81091 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:06.202630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:06.276518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069da006-7753-4e12-aeb6-83c73aa47a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:06.277297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069da006-7753-4e12-aeb6-83c73aa47a4a admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:06.277693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069da006-7753-4e12-aeb6-83c73aa47a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:06.278069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069da006-7753-4e12-aeb6-83c73aa47a4a admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:06.278902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069da006-7753-4e12-aeb6-83c73aa47a4a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:06.281753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069da006-7753-4e12-aeb6-83c73aa47a4a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:06.315032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 969/1925] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:06 2022] POST /identity/v3/users => generated 294 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:06.319893 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fb9fa9-893b-446a-88ed-c365d08e620f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:06.331683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:06.339607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fb9fa9-893b-446a-88ed-c365d08e620f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:06.354762 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bb547a-3ad4-42da-8574-6d2b8609123e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:06.355556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bb547a-3ad4-42da-8574-6d2b8609123e admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:06.355837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bb547a-3ad4-42da-8574-6d2b8609123e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:06.356107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bb547a-3ad4-42da-8574-6d2b8609123e admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:06.356801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bb547a-3ad4-42da-8574-6d2b8609123e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:06.358450 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bb547a-3ad4-42da-8574-6d2b8609123e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:06.370846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 957/1926] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:06 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:06.376235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069da006-7753-4e12-aeb6-83c73aa47a4a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:06.396246 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069da006-7753-4e12-aeb6-83c73aa47a4a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:06.404768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e8ce92-ad47-4b8f-8f1f-edfd0e6c9a8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:06.406002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e8ce92-ad47-4b8f-8f1f-edfd0e6c9a8b admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:06.406505 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e8ce92-ad47-4b8f-8f1f-edfd0e6c9a8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:06.407581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e8ce92-ad47-4b8f-8f1f-edfd0e6c9a8b admin admin] PATH_INFO: `/v3/projects/9f602d2e74544c4482ad452f3e80848d/users/cb1eb3e47453430a8038defc9e25805b/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:06.438126 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e8ce92-ad47-4b8f-8f1f-edfd0e6c9a8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f602d2e74544c4482ad452f3e80848d, user_id=cb1eb3e47453430a8038defc9e25805b, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:06.441047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e8ce92-ad47-4b8f-8f1f-edfd0e6c9a8b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:06.462554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 970/1927] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:06 2022] PUT /identity/v3/projects/9f602d2e74544c4482ad452f3e80848d/users/cb1eb3e47453430a8038defc9e25805b/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:06.463893 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bb547a-3ad4-42da-8574-6d2b8609123e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:06.475591 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bb547a-3ad4-42da-8574-6d2b8609123e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:06.551572 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298d2171-3302-4632-906b-f24ddd6d8210 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:06.552383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d2171-3302-4632-906b-f24ddd6d8210 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:06.552780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d2171-3302-4632-906b-f24ddd6d8210 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:06.553217 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d2171-3302-4632-906b-f24ddd6d8210 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:06.554310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298d2171-3302-4632-906b-f24ddd6d8210 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:06.557331 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298d2171-3302-4632-906b-f24ddd6d8210 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:06.576139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 958/1928] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:06 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:09:06.581101 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e8ce92-ad47-4b8f-8f1f-edfd0e6c9a8b admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:06.593357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e8ce92-ad47-4b8f-8f1f-edfd0e6c9a8b admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:06.599529 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc20757-6b03-445b-bebc-03b8a239e336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:06.600378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc20757-6b03-445b-bebc-03b8a239e336 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:06.600707 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc20757-6b03-445b-bebc-03b8a239e336 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:06.601126 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc20757-6b03-445b-bebc-03b8a239e336 admin admin] PATH_INFO: `/v3/projects/9f602d2e74544c4482ad452f3e80848d/users/cb1eb3e47453430a8038defc9e25805b/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:06.603461 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc20757-6b03-445b-bebc-03b8a239e336 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f602d2e74544c4482ad452f3e80848d, user_id=cb1eb3e47453430a8038defc9e25805b, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:06.606374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc20757-6b03-445b-bebc-03b8a239e336 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:06.621570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 971/1929] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:06 2022] PUT /identity/v3/projects/9f602d2e74544c4482ad452f3e80848d/users/cb1eb3e47453430a8038defc9e25805b/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:06.637310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ed7b7f-6534-4ea9-8097-3661560a26be None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:06.637840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ed7b7f-6534-4ea9-8097-3661560a26be None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:06.638220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ed7b7f-6534-4ea9-8097-3661560a26be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:06.675107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-e4ed7b7f-6534-4ea9-8097-3661560a26be None None] MFA Rules not processed for user `cb1eb3e47453430a8038defc9e25805b`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:06.676577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e4ed7b7f-6534-4ea9-8097-3661560a26be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:06.762603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 959/1930] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:06 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:06.777448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc20757-6b03-445b-bebc-03b8a239e336 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:06.779783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-1cc20757-6b03-445b-bebc-03b8a239e336 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:06.857281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc20757-6b03-445b-bebc-03b8a239e336 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:06.861129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b181bb-7d91-49ad-a278-8269fb6a6804 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:06.861961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b181bb-7d91-49ad-a278-8269fb6a6804 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:06.862243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b181bb-7d91-49ad-a278-8269fb6a6804 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:06.862517 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b181bb-7d91-49ad-a278-8269fb6a6804 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:06.934427 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b181bb-7d91-49ad-a278-8269fb6a6804 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:06.936369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b181bb-7d91-49ad-a278-8269fb6a6804 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:06.955508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 972/1931] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:06 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:09:08.258200 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fd54aa-ca09-4f49-b860-8bcc59068268 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:08.259596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fd54aa-ca09-4f49-b860-8bcc59068268 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:08.260250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fd54aa-ca09-4f49-b860-8bcc59068268 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:08.264483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 960/1932] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:09:08 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:09:08.276108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ed52a1-b8cf-452f-9e1d-fb78e39d30ca None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:08.276613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ed52a1-b8cf-452f-9e1d-fb78e39d30ca None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:08.277043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ed52a1-b8cf-452f-9e1d-fb78e39d30ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:08.368188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a7ed52a1-b8cf-452f-9e1d-fb78e39d30ca None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:08.373475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a7ed52a1-b8cf-452f-9e1d-fb78e39d30ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:08.448528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:08.584223 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 973/1933] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:09:08 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:08.713271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fd54aa-ca09-4f49-b860-8bcc59068268 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:08.721896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-73fd54aa-ca09-4f49-b860-8bcc59068268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:08.763094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fd54aa-ca09-4f49-b860-8bcc59068268 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:08.796875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8165e9b3-30d1-4dbd-8796-021ed115adaf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:08.798964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8165e9b3-30d1-4dbd-8796-021ed115adaf service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:08.798964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8165e9b3-30d1-4dbd-8796-021ed115adaf service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:08.799412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8165e9b3-30d1-4dbd-8796-021ed115adaf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:08.851276 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8165e9b3-30d1-4dbd-8796-021ed115adaf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:08.854626 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8165e9b3-30d1-4dbd-8796-021ed115adaf service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:08.878602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 961/1934] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:08 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:09:10.263470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ed52a1-b8cf-452f-9e1d-fb78e39d30ca None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:10.368446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ed52a1-b8cf-452f-9e1d-fb78e39d30ca None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:10.373155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:10.373801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:10.374107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:10.374408 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] PATH_INFO: `/v3/users/6b37ea618a944f6591d2089905b39c29` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:10.376920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b37ea618a944f6591d2089905b39c29)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:10.378830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:10.443128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b37ea618a944f6591d2089905b39c29'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:10.443794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b37ea618a944f6591d2089905b39c29'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:10.454557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b37ea618a944f6591d2089905b39c29'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:10.468239 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b37ea618a944f6591d2089905b39c29'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:10.477919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b37ea618a944f6591d2089905b39c29'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:10.484517 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 974/1935] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:10 2022] DELETE /identity/v3/users/6b37ea618a944f6591d2089905b39c29 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:10.489526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc47439a-72a1-4b49-91f8-62f1e3bf6f07 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:10.490015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc47439a-72a1-4b49-91f8-62f1e3bf6f07 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:10.490117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc47439a-72a1-4b49-91f8-62f1e3bf6f07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:10.513261 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-cc47439a-72a1-4b49-91f8-62f1e3bf6f07 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:10.514677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-cc47439a-72a1-4b49-91f8-62f1e3bf6f07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:10.606409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 962/1936] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:10.617609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:10.619916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:10.693692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:10.711856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2134e0de-838f-4444-9f29-07ad63479d30 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:10.717315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0531421e-2e40-4b17-9bb0-bd1b63408e2e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:10.718140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0531421e-2e40-4b17-9bb0-bd1b63408e2e service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:10.718771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0531421e-2e40-4b17-9bb0-bd1b63408e2e service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:10.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0531421e-2e40-4b17-9bb0-bd1b63408e2e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:10.732857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0531421e-2e40-4b17-9bb0-bd1b63408e2e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:10.734474 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0531421e-2e40-4b17-9bb0-bd1b63408e2e service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:10.748156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 975/1937] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:09:11.294093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc47439a-72a1-4b49-91f8-62f1e3bf6f07 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:11.297463 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-cc47439a-72a1-4b49-91f8-62f1e3bf6f07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:11.317108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:11.324384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc47439a-72a1-4b49-91f8-62f1e3bf6f07 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:11.346955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:11.347672 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:11.347957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:11.348232 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] PATH_INFO: `/v3/projects/4989caebd6cf4d7da7278b00ef14ef50` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:11.351541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4989caebd6cf4d7da7278b00ef14ef50)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:11.353325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:11.490734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4989caebd6cf4d7da7278b00ef14ef50', 'request_id': 'req-30b20a2e-6469-4c2d-a2d8-44cb4e033815'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:11.503175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4989caebd6cf4d7da7278b00ef14ef50', 'request_id': 'req-30b20a2e-6469-4c2d-a2d8-44cb4e033815'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:11.515098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4989caebd6cf4d7da7278b00ef14ef50', 'request_id': 'req-30b20a2e-6469-4c2d-a2d8-44cb4e033815'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:11.561087 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] The token cache is being invalidate because project 4989caebd6cf4d7da7278b00ef14ef50 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:11.561546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:11.564167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:11.567777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 963/1938] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:11 2022] DELETE /identity/v3/projects/4989caebd6cf4d7da7278b00ef14ef50 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:11.572450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0531421e-2e40-4b17-9bb0-bd1b63408e2e service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:11.596042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0531421e-2e40-4b17-9bb0-bd1b63408e2e service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:11.678678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:11.680322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:11.715497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:11.717570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:11.718081 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:11.718580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] PATH_INFO: `/v3/users/cb1eb3e47453430a8038defc9e25805b` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:11.723175 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb1eb3e47453430a8038defc9e25805b)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:11.726199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:11.771865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b20a2e-6469-4c2d-a2d8-44cb4e033815 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:11.778835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3681f7-69a5-4b6d-8d80-c533a3c8379d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:11.779868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3681f7-69a5-4b6d-8d80-c533a3c8379d service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:11.780336 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3681f7-69a5-4b6d-8d80-c533a3c8379d service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:11.780617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3681f7-69a5-4b6d-8d80-c533a3c8379d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:11.782706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ba3681f7-69a5-4b6d-8d80-c533a3c8379d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:11.805273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb1eb3e47453430a8038defc9e25805b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:11.806150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb1eb3e47453430a8038defc9e25805b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:11.817965 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb1eb3e47453430a8038defc9e25805b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:11.837215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb1eb3e47453430a8038defc9e25805b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:11.847939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb1eb3e47453430a8038defc9e25805b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:11.856722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 976/1939] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:11 2022] DELETE /identity/v3/users/cb1eb3e47453430a8038defc9e25805b => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:11.877708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3681f7-69a5-4b6d-8d80-c533a3c8379d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:11.879228 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3681f7-69a5-4b6d-8d80-c533a3c8379d service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:11.890357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 964/1940] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:11 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:09:12.260058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:12.262558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:12.446609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744fc597-5dc9-4dcb-b193-5ba8d340583c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:12.458425 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:12.459933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:12.460539 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:12.461039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] PATH_INFO: `/v3/projects/9f602d2e74544c4482ad452f3e80848d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:12.464008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f602d2e74544c4482ad452f3e80848d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:12.467610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:12.484374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:12.596644 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f602d2e74544c4482ad452f3e80848d', 'request_id': 'req-498b717d-381c-475d-bbe2-66f377db0226'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:12.605900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f602d2e74544c4482ad452f3e80848d', 'request_id': 'req-498b717d-381c-475d-bbe2-66f377db0226'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:12.621333 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f602d2e74544c4482ad452f3e80848d', 'request_id': 'req-498b717d-381c-475d-bbe2-66f377db0226'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:12.638393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] The token cache is being invalidate because project 9f602d2e74544c4482ad452f3e80848d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:12.639015 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:12.640036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:12.642639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 977/1941] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:12 2022] DELETE /identity/v3/projects/9f602d2e74544c4482ad452f3e80848d => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:12.650048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553b0172-42a2-478f-b0a2-d5180a47592e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:12.651271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553b0172-42a2-478f-b0a2-d5180a47592e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:12.651809 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553b0172-42a2-478f-b0a2-d5180a47592e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:12.667564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:12.671053 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:12.743782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-553b0172-42a2-478f-b0a2-d5180a47592e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:12.745079 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-553b0172-42a2-478f-b0a2-d5180a47592e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:12.864783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:12.948648 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 965/1942] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:12.967095 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae454d2d-92ed-4a54-b947-0e35a474c92a None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:12.968820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae454d2d-92ed-4a54-b947-0e35a474c92a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:12.971151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae454d2d-92ed-4a54-b947-0e35a474c92a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:12.976349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498b717d-381c-475d-bbe2-66f377db0226 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:12.997395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e377bf6e-e649-4c35-b2ee-82e81824b6f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:12.998758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e377bf6e-e649-4c35-b2ee-82e81824b6f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:12.999036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e377bf6e-e649-4c35-b2ee-82e81824b6f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:12.999310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e377bf6e-e649-4c35-b2ee-82e81824b6f2 admin admin] PATH_INFO: `/v3/users/79b46b24aade43c5b0c3821de4cfd3fe` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:13.002681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e377bf6e-e649-4c35-b2ee-82e81824b6f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=79b46b24aade43c5b0c3821de4cfd3fe)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:13.007225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ae454d2d-92ed-4a54-b947-0e35a474c92a None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:13.009546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ae454d2d-92ed-4a54-b947-0e35a474c92a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:13.011524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e377bf6e-e649-4c35-b2ee-82e81824b6f2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:13.052913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 966/1943] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:13.054600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae454d2d-92ed-4a54-b947-0e35a474c92a None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:13.074216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae454d2d-92ed-4a54-b947-0e35a474c92a None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:13.086773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf39d3d-6ea5-4ec9-ae0d-d3e738a432ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:13.089274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf39d3d-6ea5-4ec9-ae0d-d3e738a432ef admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:13.089620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf39d3d-6ea5-4ec9-ae0d-d3e738a432ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:13.089902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf39d3d-6ea5-4ec9-ae0d-d3e738a432ef admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:13.090653 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf39d3d-6ea5-4ec9-ae0d-d3e738a432ef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:13.090960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf39d3d-6ea5-4ec9-ae0d-d3e738a432ef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:13.097807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf39d3d-6ea5-4ec9-ae0d-d3e738a432ef admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:13.129899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 967/1944] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:09:13.135226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf39d3d-6ea5-4ec9-ae0d-d3e738a432ef admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:13.154488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf39d3d-6ea5-4ec9-ae0d-d3e738a432ef admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:13.164557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0e49ed-3f55-4276-91e0-f536bb5dc315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:13.165591 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0e49ed-3f55-4276-91e0-f536bb5dc315 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:13.166078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0e49ed-3f55-4276-91e0-f536bb5dc315 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:13.166560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0e49ed-3f55-4276-91e0-f536bb5dc315 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:13.167819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0e49ed-3f55-4276-91e0-f536bb5dc315 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:13.171671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0e49ed-3f55-4276-91e0-f536bb5dc315 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:13.226704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e377bf6e-e649-4c35-b2ee-82e81824b6f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79b46b24aade43c5b0c3821de4cfd3fe'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:13.228653 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e377bf6e-e649-4c35-b2ee-82e81824b6f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79b46b24aade43c5b0c3821de4cfd3fe'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:13.244314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e377bf6e-e649-4c35-b2ee-82e81824b6f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79b46b24aade43c5b0c3821de4cfd3fe'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:13.257523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 968/1945] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:13 2022] POST /identity/v3/projects => generated 363 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:13.258826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0e49ed-3f55-4276-91e0-f536bb5dc315 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:13.271151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e377bf6e-e649-4c35-b2ee-82e81824b6f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79b46b24aade43c5b0c3821de4cfd3fe'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:13.287115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e377bf6e-e649-4c35-b2ee-82e81824b6f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79b46b24aade43c5b0c3821de4cfd3fe'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:13.300450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 978/1946] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:12 2022] DELETE /identity/v3/users/79b46b24aade43c5b0c3821de4cfd3fe => generated 0 bytes in 630 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:13.302950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e84b3b-d660-4046-b014-39336e813b04 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:13.303318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e84b3b-d660-4046-b014-39336e813b04 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:13.303824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e84b3b-d660-4046-b014-39336e813b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:13.353620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b9e84b3b-d660-4046-b014-39336e813b04 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:13.355916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b9e84b3b-d660-4046-b014-39336e813b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:13.367146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0e49ed-3f55-4276-91e0-f536bb5dc315 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:13.388850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:13.473208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6b2b19-e641-4f8c-9d72-e382452466d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:13.474453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6b2b19-e641-4f8c-9d72-e382452466d5 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:13.475061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6b2b19-e641-4f8c-9d72-e382452466d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:13.475409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6b2b19-e641-4f8c-9d72-e382452466d5 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:13.476835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6b2b19-e641-4f8c-9d72-e382452466d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:13.480998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6b2b19-e641-4f8c-9d72-e382452466d5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:13.495393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 979/1947] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:13.517021 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e84b3b-d660-4046-b014-39336e813b04 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:13.534240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b9e84b3b-d660-4046-b014-39336e813b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:13.550066 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 969/1948] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:13 2022] POST /identity/v3/users => generated 282 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:09:13.551367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6b2b19-e641-4f8c-9d72-e382452466d5 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:13.558158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8b6b2b19-e641-4f8c-9d72-e382452466d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:13.579641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6b2b19-e641-4f8c-9d72-e382452466d5 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:13.585551 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eb09e3-b5fb-4470-802d-d0b442f17c08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:13.586273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eb09e3-b5fb-4470-802d-d0b442f17c08 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:13.586535 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eb09e3-b5fb-4470-802d-d0b442f17c08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:13.586812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eb09e3-b5fb-4470-802d-d0b442f17c08 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:13.587978 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eb09e3-b5fb-4470-802d-d0b442f17c08 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:13.589767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eb09e3-b5fb-4470-802d-d0b442f17c08 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:13.606653 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 970/1949] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:13 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:13.608234 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eb09e3-b5fb-4470-802d-d0b442f17c08 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:13.627419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eb09e3-b5fb-4470-802d-d0b442f17c08 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:13.630432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb42677-1681-4353-82d7-f6dfecd8a592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:13.631242 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb42677-1681-4353-82d7-f6dfecd8a592 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:13.631571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb42677-1681-4353-82d7-f6dfecd8a592 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:13.631878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb42677-1681-4353-82d7-f6dfecd8a592 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:13.632869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb42677-1681-4353-82d7-f6dfecd8a592 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:13.634761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb42677-1681-4353-82d7-f6dfecd8a592 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:13.657200 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 971/1950] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:13 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:13.659851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb42677-1681-4353-82d7-f6dfecd8a592 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:13.674635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb42677-1681-4353-82d7-f6dfecd8a592 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:13.680316 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ceb414a-f306-443b-adba-8497b02e68b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:13.681771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceb414a-f306-443b-adba-8497b02e68b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:13.682224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceb414a-f306-443b-adba-8497b02e68b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:13.682667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceb414a-f306-443b-adba-8497b02e68b5 admin admin] PATH_INFO: `/v3/projects/9534f49685b24d218e97834be3980c68/users/6febcd137bf44b9b8cd32deec0896f74/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:13.695844 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e84b3b-d660-4046-b014-39336e813b04 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:13.701241 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:13.708257 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ceb414a-f306-443b-adba-8497b02e68b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9534f49685b24d218e97834be3980c68, user_id=6febcd137bf44b9b8cd32deec0896f74, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:13.708489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45f844c-9145-4329-9c26-55fd0892544e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:13.710335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45f844c-9145-4329-9c26-55fd0892544e service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:13.711524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45f844c-9145-4329-9c26-55fd0892544e service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:13.712170 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45f844c-9145-4329-9c26-55fd0892544e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:13.713314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ceb414a-f306-443b-adba-8497b02e68b5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:13.740090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45f844c-9145-4329-9c26-55fd0892544e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:13.745613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45f844c-9145-4329-9c26-55fd0892544e service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:13.751814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 972/1951] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:13 2022] PUT /identity/v3/projects/9534f49685b24d218e97834be3980c68/users/6febcd137bf44b9b8cd32deec0896f74/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:13.753997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acba01bd-5923-4f6e-851d-7981c5b8dd25 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:13.754288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acba01bd-5923-4f6e-851d-7981c5b8dd25 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:13.755288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acba01bd-5923-4f6e-851d-7981c5b8dd25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:13.780538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 980/1952] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:13.801401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-acba01bd-5923-4f6e-851d-7981c5b8dd25 None None] MFA Rules not processed for user `6febcd137bf44b9b8cd32deec0896f74`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:13.803739 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-acba01bd-5923-4f6e-851d-7981c5b8dd25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:13.940686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 973/1953] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:13 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:09:13.955336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45f844c-9145-4329-9c26-55fd0892544e service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:13.957294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c45f844c-9145-4329-9c26-55fd0892544e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:14.051768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45f844c-9145-4329-9c26-55fd0892544e service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:14.121919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e001d9bb-b149-4c1f-ab60-bfcd315e62fc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:14.122567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e001d9bb-b149-4c1f-ab60-bfcd315e62fc service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:14.123660 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e001d9bb-b149-4c1f-ab60-bfcd315e62fc service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:14.123660 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e001d9bb-b149-4c1f-ab60-bfcd315e62fc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:14.136093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e001d9bb-b149-4c1f-ab60-bfcd315e62fc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:14.137795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e001d9bb-b149-4c1f-ab60-bfcd315e62fc service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:14.150882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 981/1954] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:13 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:09:14.472841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8043090b-02e1-4ef5-96d5-36f0b485fd30 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:14.473581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8043090b-02e1-4ef5-96d5-36f0b485fd30 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:14.474053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8043090b-02e1-4ef5-96d5-36f0b485fd30 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:14.476594 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 974/1955] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:09:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:09:14.914496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6f32bc-56f9-4114-9c10-33a85db67f40 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:14.915318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6f32bc-56f9-4114-9c10-33a85db67f40 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:14.915318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6f32bc-56f9-4114-9c10-33a85db67f40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:14.978044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ed6f32bc-56f9-4114-9c10-33a85db67f40 None None] MFA Rules not processed for user `1d562ccd50e4492281a9f435f809d49b`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:14.980682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ed6f32bc-56f9-4114-9c10-33a85db67f40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:15.003155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:15.133565 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 982/1956] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:09:14 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:15.174571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8043090b-02e1-4ef5-96d5-36f0b485fd30 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:15.176721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8043090b-02e1-4ef5-96d5-36f0b485fd30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:15.216333 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8043090b-02e1-4ef5-96d5-36f0b485fd30 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:15.225620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d6cc96-6572-4f8c-8128-24c4c2091153 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:15.226831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d6cc96-6572-4f8c-8128-24c4c2091153 service placement] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:15.227518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d6cc96-6572-4f8c-8128-24c4c2091153 service placement] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:15.228050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d6cc96-6572-4f8c-8128-24c4c2091153 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:15.265605 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d6cc96-6572-4f8c-8128-24c4c2091153 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:15.270185 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d6cc96-6572-4f8c-8128-24c4c2091153 service placement] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:15.291726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 975/1957] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:15 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:15.693193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6f32bc-56f9-4114-9c10-33a85db67f40 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:15.696096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ed6f32bc-56f9-4114-9c10-33a85db67f40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:15.701659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d6cc96-6572-4f8c-8128-24c4c2091153 service placement] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:15.703287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c2d6cc96-6572-4f8c-8128-24c4c2091153 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:15.846923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6f32bc-56f9-4114-9c10-33a85db67f40 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:15.854853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:15.855800 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:15.856526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:15.856739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] PATH_INFO: `/v3/projects/c57d09c6a1c84fd492393aebb2101715` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:15.859015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d6cc96-6572-4f8c-8128-24c4c2091153 service placement] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:15.860968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c57d09c6a1c84fd492393aebb2101715)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:15.863881 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:15.874365 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712bcce2-7e4f-44da-aa95-1204a6b317a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:15.875047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712bcce2-7e4f-44da-aa95-1204a6b317a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:15.876032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712bcce2-7e4f-44da-aa95-1204a6b317a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:15.876317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712bcce2-7e4f-44da-aa95-1204a6b317a7 admin admin] PATH_INFO: `/v3/users/44e3542b1dd5485e89c1804bc6467462` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:15.878222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712bcce2-7e4f-44da-aa95-1204a6b317a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44e3542b1dd5485e89c1804bc6467462)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:15.880243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712bcce2-7e4f-44da-aa95-1204a6b317a7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:16.006719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c57d09c6a1c84fd492393aebb2101715', 'request_id': 'req-ace7d25a-e58b-44a4-9e38-8b9e01790481'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.020509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c57d09c6a1c84fd492393aebb2101715', 'request_id': 'req-ace7d25a-e58b-44a4-9e38-8b9e01790481'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.043084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c57d09c6a1c84fd492393aebb2101715', 'request_id': 'req-ace7d25a-e58b-44a4-9e38-8b9e01790481'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.063650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-712bcce2-7e4f-44da-aa95-1204a6b317a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44e3542b1dd5485e89c1804bc6467462'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.076648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] The token cache is being invalidate because project c57d09c6a1c84fd492393aebb2101715 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:16.077142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.083122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.084945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-712bcce2-7e4f-44da-aa95-1204a6b317a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44e3542b1dd5485e89c1804bc6467462'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.086514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-712bcce2-7e4f-44da-aa95-1204a6b317a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44e3542b1dd5485e89c1804bc6467462'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.103409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=91,keystone:INSERT=1,keystone:DELETE=8 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:16.108849 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 983/1958] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:15 2022] DELETE /identity/v3/projects/c57d09c6a1c84fd492393aebb2101715 => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:16.113225 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:16.116895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-712bcce2-7e4f-44da-aa95-1204a6b317a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44e3542b1dd5485e89c1804bc6467462'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.118622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-712bcce2-7e4f-44da-aa95-1204a6b317a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44e3542b1dd5485e89c1804bc6467462'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.124886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:16.145472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 976/1959] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:15 2022] DELETE /identity/v3/users/44e3542b1dd5485e89c1804bc6467462 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:16.149826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdfd65a-8cbb-4a4b-953c-32d99986e368 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:16.149826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdfd65a-8cbb-4a4b-953c-32d99986e368 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:16.149826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdfd65a-8cbb-4a4b-953c-32d99986e368 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:16.240832 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-4bdfd65a-8cbb-4a4b-953c-32d99986e368 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:16.244061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-4bdfd65a-8cbb-4a4b-953c-32d99986e368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:16.368062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:16.425219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 977/1960] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:16.427418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac69852-3c88-4779-8fe7-98ad812bea72 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:16.427933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac69852-3c88-4779-8fe7-98ad812bea72 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:16.428700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac69852-3c88-4779-8fe7-98ad812bea72 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:16.428902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace7d25a-e58b-44a4-9e38-8b9e01790481 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:16.432504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 978/1961] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:09:16 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:09:16.448514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa84f50-b92c-4f55-acd1-77a016527640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:16.449228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa84f50-b92c-4f55-acd1-77a016527640 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:16.449511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa84f50-b92c-4f55-acd1-77a016527640 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:16.449781 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa84f50-b92c-4f55-acd1-77a016527640 admin admin] PATH_INFO: `/v3/users/e6d9332df3fc4b3a8d2b7b8a738521db` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:16.455597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa84f50-b92c-4f55-acd1-77a016527640 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e6d9332df3fc4b3a8d2b7b8a738521db)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:16.458216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa84f50-b92c-4f55-acd1-77a016527640 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:16.471878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac69852-3c88-4779-8fe7-98ad812bea72 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:16.480655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5ac69852-3c88-4779-8fe7-98ad812bea72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:16.662737 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6aa84f50-b92c-4f55-acd1-77a016527640 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e6d9332df3fc4b3a8d2b7b8a738521db'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.663220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6aa84f50-b92c-4f55-acd1-77a016527640 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e6d9332df3fc4b3a8d2b7b8a738521db'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.692299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6aa84f50-b92c-4f55-acd1-77a016527640 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e6d9332df3fc4b3a8d2b7b8a738521db'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.696199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac69852-3c88-4779-8fe7-98ad812bea72 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:16.705414 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cbd842-aa2e-4491-a77d-43f18ec912ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:16.707155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cbd842-aa2e-4491-a77d-43f18ec912ac service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:16.708372 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cbd842-aa2e-4491-a77d-43f18ec912ac service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:16.710469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cbd842-aa2e-4491-a77d-43f18ec912ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:16.716099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6aa84f50-b92c-4f55-acd1-77a016527640 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e6d9332df3fc4b3a8d2b7b8a738521db'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.736940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cbd842-aa2e-4491-a77d-43f18ec912ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:16.738417 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cbd842-aa2e-4491-a77d-43f18ec912ac service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:16.739270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6aa84f50-b92c-4f55-acd1-77a016527640 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e6d9332df3fc4b3a8d2b7b8a738521db'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:16.756427 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 984/1962] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:16 2022] DELETE /identity/v3/users/e6d9332df3fc4b3a8d2b7b8a738521db => generated 0 bytes in 646 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:16.758285 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff149b41-cad1-4bc7-9d8c-2166284c34ca None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:16.758866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff149b41-cad1-4bc7-9d8c-2166284c34ca None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:16.759402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff149b41-cad1-4bc7-9d8c-2166284c34ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:16.779604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 979/1963] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:16.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ff149b41-cad1-4bc7-9d8c-2166284c34ca None None] MFA Rules not processed for user `1d562ccd50e4492281a9f435f809d49b`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:16.819620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ff149b41-cad1-4bc7-9d8c-2166284c34ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:16.852320 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cbd842-aa2e-4491-a77d-43f18ec912ac service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:16.853924 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b0cbd842-aa2e-4491-a77d-43f18ec912ac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:16.879102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cbd842-aa2e-4491-a77d-43f18ec912ac service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:17.000083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 985/1964] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:09:16 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:17.027958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff149b41-cad1-4bc7-9d8c-2166284c34ca None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:17.031395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ff149b41-cad1-4bc7-9d8c-2166284c34ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:17.051949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:17.053450 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:17.053944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:17.054456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] PATH_INFO: `/v3/users/bc8c26db74024f049674334e3519d3f5` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:17.056527 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc8c26db74024f049674334e3519d3f5)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:17.060524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:17.062931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff149b41-cad1-4bc7-9d8c-2166284c34ca None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:17.077278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b10f98-8f2d-4c8f-a085-92e7d2087b31 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:17.078955 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b10f98-8f2d-4c8f-a085-92e7d2087b31 service placement] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:17.079569 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b10f98-8f2d-4c8f-a085-92e7d2087b31 service placement] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:17.080148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b10f98-8f2d-4c8f-a085-92e7d2087b31 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:17.101727 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b10f98-8f2d-4c8f-a085-92e7d2087b31 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:17.109046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b10f98-8f2d-4c8f-a085-92e7d2087b31 service placement] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:17.132147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:17.144251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 986/1965] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:17 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:17.173206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc8c26db74024f049674334e3519d3f5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.210810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc8c26db74024f049674334e3519d3f5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.211677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc8c26db74024f049674334e3519d3f5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.261898 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc8c26db74024f049674334e3519d3f5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.263262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc8c26db74024f049674334e3519d3f5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.294572 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 980/1966] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:16 2022] DELETE /identity/v3/users/bc8c26db74024f049674334e3519d3f5 => generated 0 bytes in 443 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:17.501887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b10f98-8f2d-4c8f-a085-92e7d2087b31 service placement] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:17.505017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-71b10f98-8f2d-4c8f-a085-92e7d2087b31 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:17.548054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b10f98-8f2d-4c8f-a085-92e7d2087b31 service placement] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:17.617201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:17.622650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:17.683856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:17.709700 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:17.710633 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:17.711134 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:17.711640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] PATH_INFO: `/v3/projects/353d268d38764c48a252cdf9a5146156` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:17.714109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] RBAC: Authorizing `identity:delete_project(project_id=353d268d38764c48a252cdf9a5146156)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:17.716116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:17.769852 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d3e9f6-c8ae-45c6-b0b2-d7d86d85dd71 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:17.775428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236a0085-38da-4fd8-998c-156f7cddb284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:17.776111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236a0085-38da-4fd8-998c-156f7cddb284 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:17.776394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236a0085-38da-4fd8-998c-156f7cddb284 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:17.776706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236a0085-38da-4fd8-998c-156f7cddb284 admin admin] PATH_INFO: `/v3/projects/bd337d5066d649d7a1233e454bf73c3e` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:17.778041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236a0085-38da-4fd8-998c-156f7cddb284 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd337d5066d649d7a1233e454bf73c3e)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:17.780287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236a0085-38da-4fd8-998c-156f7cddb284 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:17.861441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '353d268d38764c48a252cdf9a5146156', 'request_id': 'req-cb70de52-e446-4180-8e11-d1c4ee241711'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.878291 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-236a0085-38da-4fd8-998c-156f7cddb284 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd337d5066d649d7a1233e454bf73c3e', 'request_id': 'req-236a0085-38da-4fd8-998c-156f7cddb284'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.878790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '353d268d38764c48a252cdf9a5146156', 'request_id': 'req-cb70de52-e446-4180-8e11-d1c4ee241711'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.893488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '353d268d38764c48a252cdf9a5146156', 'request_id': 'req-cb70de52-e446-4180-8e11-d1c4ee241711'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.895659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-236a0085-38da-4fd8-998c-156f7cddb284 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd337d5066d649d7a1233e454bf73c3e', 'request_id': 'req-236a0085-38da-4fd8-998c-156f7cddb284'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.912524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] The token cache is being invalidate because project 353d268d38764c48a252cdf9a5146156 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:17.912795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.913537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.916650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 987/1967] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:17 2022] DELETE /identity/v3/projects/353d268d38764c48a252cdf9a5146156 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:17.917741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:17.919321 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:17.935785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-236a0085-38da-4fd8-998c-156f7cddb284 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd337d5066d649d7a1233e454bf73c3e', 'request_id': 'req-236a0085-38da-4fd8-998c-156f7cddb284'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.979637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-236a0085-38da-4fd8-998c-156f7cddb284 admin admin] The token cache is being invalidate because project bd337d5066d649d7a1233e454bf73c3e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:17.982353 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-236a0085-38da-4fd8-998c-156f7cddb284 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:17.987529 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-236a0085-38da-4fd8-998c-156f7cddb284 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.003776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 981/1968] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:17 2022] DELETE /identity/v3/projects/bd337d5066d649d7a1233e454bf73c3e => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:18.003776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d0f0b0-a1ac-4566-9f59-2edf5732daa9 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:18.003776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d0f0b0-a1ac-4566-9f59-2edf5732daa9 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:18.003776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d0f0b0-a1ac-4566-9f59-2edf5732daa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:18.028144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb70de52-e446-4180-8e11-d1c4ee241711 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:18.040662 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-f7d0f0b0-a1ac-4566-9f59-2edf5732daa9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:18.042400 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f7d0f0b0-a1ac-4566-9f59-2edf5732daa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:18.089738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:18.124406 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:18.125692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:18.126287 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:18.126805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811 admin admin] PATH_INFO: `/v3/projects/9afc8112c0d54bdf987ef0c44d572471` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:18.129177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9afc8112c0d54bdf987ef0c44d572471)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:18.133045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:18.157935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 982/1969] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:18.159240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d0f0b0-a1ac-4566-9f59-2edf5732daa9 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:18.160895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f7d0f0b0-a1ac-4566-9f59-2edf5732daa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:18.180141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d0f0b0-a1ac-4566-9f59-2edf5732daa9 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:18.187100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:18.188265 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:18.188603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:18.189081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] PATH_INFO: `/v3/users/376158b98a414cf78dceb70ba4f357c9` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:18.191338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] RBAC: Authorizing `identity:delete_user(user_id=376158b98a414cf78dceb70ba4f357c9)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:18.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:18.206052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:18.249460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9afc8112c0d54bdf987ef0c44d572471', 'request_id': 'req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.261595 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9afc8112c0d54bdf987ef0c44d572471', 'request_id': 'req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.273356 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '376158b98a414cf78dceb70ba4f357c9'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.277435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9afc8112c0d54bdf987ef0c44d572471', 'request_id': 'req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.288683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '376158b98a414cf78dceb70ba4f357c9'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.290075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '376158b98a414cf78dceb70ba4f357c9'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.298181 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811 admin admin] The token cache is being invalidate because project 9afc8112c0d54bdf987ef0c44d572471 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:18.298834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.300281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0112d3f9-4a7d-4cbb-9915-8a9671bfd811 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.314130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 988/1970] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:17 2022] DELETE /identity/v3/projects/9afc8112c0d54bdf987ef0c44d572471 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:18.316394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47dab51-5e35-4dbe-ad01-6e34d4d267e4 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:18.317125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47dab51-5e35-4dbe-ad01-6e34d4d267e4 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:18.318549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47dab51-5e35-4dbe-ad01-6e34d4d267e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:18.318698 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '376158b98a414cf78dceb70ba4f357c9'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.320128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '376158b98a414cf78dceb70ba4f357c9'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.350244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-e47dab51-5e35-4dbe-ad01-6e34d4d267e4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:18.350621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 983/1971] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:18 2022] DELETE /identity/v3/users/376158b98a414cf78dceb70ba4f357c9 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:18.351727 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e47dab51-5e35-4dbe-ad01-6e34d4d267e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:18.352761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:18.356157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:18.454191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 989/1972] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:18.459704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47dab51-5e35-4dbe-ad01-6e34d4d267e4 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:18.481392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47dab51-5e35-4dbe-ad01-6e34d4d267e4 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:18.490689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14a99ad-5e0a-445a-b543-b75d4e5d2153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:18.492086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14a99ad-5e0a-445a-b543-b75d4e5d2153 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:18.492625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14a99ad-5e0a-445a-b543-b75d4e5d2153 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:18.493267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14a99ad-5e0a-445a-b543-b75d4e5d2153 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:18.494522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14a99ad-5e0a-445a-b543-b75d4e5d2153 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:18.495096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14a99ad-5e0a-445a-b543-b75d4e5d2153 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:18.498928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14a99ad-5e0a-445a-b543-b75d4e5d2153 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:18.506733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53826613-c35d-40c2-ac63-79d117f71f6c admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:18.513018 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:18.513694 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:18.514032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:18.514310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] PATH_INFO: `/v3/users/4318271b271749aeaaa74adee2651b8c` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:18.517858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=4318271b271749aeaaa74adee2651b8c)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:18.520204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:18.524421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 990/1973] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:09:18.531296 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14a99ad-5e0a-445a-b543-b75d4e5d2153 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:18.551682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14a99ad-5e0a-445a-b543-b75d4e5d2153 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:18.568274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea64939-3af9-43b8-ba18-6b185700fa47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:18.568968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea64939-3af9-43b8-ba18-6b185700fa47 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:18.569250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea64939-3af9-43b8-ba18-6b185700fa47 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:18.569592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea64939-3af9-43b8-ba18-6b185700fa47 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:18.570377 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea64939-3af9-43b8-ba18-6b185700fa47 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:18.575069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea64939-3af9-43b8-ba18-6b185700fa47 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:18.615265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 991/1974] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:18 2022] POST /identity/v3/projects => generated 391 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:18.620191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea64939-3af9-43b8-ba18-6b185700fa47 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:18.629725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4318271b271749aeaaa74adee2651b8c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.642026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:18.644684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4318271b271749aeaaa74adee2651b8c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.646907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4318271b271749aeaaa74adee2651b8c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.667722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4318271b271749aeaaa74adee2651b8c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.671784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4318271b271749aeaaa74adee2651b8c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:18.715493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 984/1975] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:18 2022] DELETE /identity/v3/users/4318271b271749aeaaa74adee2651b8c => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:18.719175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:18.721643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:18.783115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea64939-3af9-43b8-ba18-6b185700fa47 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:18.849892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:18.907632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c8c5e4-8103-4331-9417-73662edfe175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:18.909468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c8c5e4-8103-4331-9417-73662edfe175 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:18.910021 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c8c5e4-8103-4331-9417-73662edfe175 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:18.910800 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c8c5e4-8103-4331-9417-73662edfe175 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:18.911809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c8c5e4-8103-4331-9417-73662edfe175 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:18.915544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c8c5e4-8103-4331-9417-73662edfe175 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:18.952854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a17b18-6606-40ef-b56c-3ba31bb397dc admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:18.957056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3acce2-a44c-496a-8da1-66dd40ae343c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:18.957741 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3acce2-a44c-496a-8da1-66dd40ae343c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:18.958050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3acce2-a44c-496a-8da1-66dd40ae343c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:18.958289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3acce2-a44c-496a-8da1-66dd40ae343c admin admin] PATH_INFO: `/v3/projects/5baf560f78b446f691b16d77fdc05407` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:18.958820 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 992/1976] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:18 2022] POST /identity/v3/users => generated 296 bytes in 339 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:18.959853 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3acce2-a44c-496a-8da1-66dd40ae343c admin admin] RBAC: Authorizing `identity:delete_project(project_id=5baf560f78b446f691b16d77fdc05407)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:18.962516 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3acce2-a44c-496a-8da1-66dd40ae343c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:18.964243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c8c5e4-8103-4331-9417-73662edfe175 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:18.965986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b8c8c5e4-8103-4331-9417-73662edfe175 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:18.979502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c8c5e4-8103-4331-9417-73662edfe175 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:18.985212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf84b93-20ea-4397-a4da-6cff94b84ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:18.985858 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf84b93-20ea-4397-a4da-6cff94b84ee7 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:18.986136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf84b93-20ea-4397-a4da-6cff94b84ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:18.986407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf84b93-20ea-4397-a4da-6cff94b84ee7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:18.988306 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf84b93-20ea-4397-a4da-6cff94b84ee7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:18.990026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf84b93-20ea-4397-a4da-6cff94b84ee7 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:19.003973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 993/1977] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:18 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:19.008276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf84b93-20ea-4397-a4da-6cff94b84ee7 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:19.036244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf84b93-20ea-4397-a4da-6cff94b84ee7 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:19.045733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-3c3acce2-a44c-496a-8da1-66dd40ae343c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5baf560f78b446f691b16d77fdc05407', 'request_id': 'req-3c3acce2-a44c-496a-8da1-66dd40ae343c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.056183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-3c3acce2-a44c-496a-8da1-66dd40ae343c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5baf560f78b446f691b16d77fdc05407', 'request_id': 'req-3c3acce2-a44c-496a-8da1-66dd40ae343c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.075860 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-3c3acce2-a44c-496a-8da1-66dd40ae343c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5baf560f78b446f691b16d77fdc05407', 'request_id': 'req-3c3acce2-a44c-496a-8da1-66dd40ae343c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.104690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-3c3acce2-a44c-496a-8da1-66dd40ae343c admin admin] The token cache is being invalidate because project 5baf560f78b446f691b16d77fdc05407 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:19.105145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-3c3acce2-a44c-496a-8da1-66dd40ae343c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.106246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-3c3acce2-a44c-496a-8da1-66dd40ae343c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.109256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 985/1978] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:18 2022] DELETE /identity/v3/projects/5baf560f78b446f691b16d77fdc05407 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:19.115602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e68c38-662c-4860-b223-d492d47ffb04 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:19.116265 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e68c38-662c-4860-b223-d492d47ffb04 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:19.116824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e68c38-662c-4860-b223-d492d47ffb04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:19.159017 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a8e68c38-662c-4860-b223-d492d47ffb04 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:19.162244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a8e68c38-662c-4860-b223-d492d47ffb04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:19.167904 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f76544-7e24-42bd-bdd7-ab8230818407 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:19.169179 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f76544-7e24-42bd-bdd7-ab8230818407 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:19.169752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f76544-7e24-42bd-bdd7-ab8230818407 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:19.170231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f76544-7e24-42bd-bdd7-ab8230818407 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:19.178172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f76544-7e24-42bd-bdd7-ab8230818407 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:19.182046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f76544-7e24-42bd-bdd7-ab8230818407 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:19.185661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 986/1979] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:19.197173 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74a62f1-982f-468c-bbbe-2cbf55e10471 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:19.198402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74a62f1-982f-468c-bbbe-2cbf55e10471 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:19.199399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74a62f1-982f-468c-bbbe-2cbf55e10471 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:19.207297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 994/1980] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:19 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:19.209449 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f76544-7e24-42bd-bdd7-ab8230818407 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:19.214110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-47f76544-7e24-42bd-bdd7-ab8230818407 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:19.245061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f76544-7e24-42bd-bdd7-ab8230818407 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:19.250837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d74a62f1-982f-468c-bbbe-2cbf55e10471 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:19.253527 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d74a62f1-982f-468c-bbbe-2cbf55e10471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:19.259048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10773a1-2788-4fe5-9c6d-5536c70cb902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:19.259048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10773a1-2788-4fe5-9c6d-5536c70cb902 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:19.259048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10773a1-2788-4fe5-9c6d-5536c70cb902 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:19.259704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10773a1-2788-4fe5-9c6d-5536c70cb902 admin admin] PATH_INFO: `/v3/projects/8d82f770c9534c14b4cec601d5dd68b9` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:19.263235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10773a1-2788-4fe5-9c6d-5536c70cb902 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d82f770c9534c14b4cec601d5dd68b9)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:19.267169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10773a1-2788-4fe5-9c6d-5536c70cb902 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:19.272195 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 987/1981] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:19.275726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74a62f1-982f-468c-bbbe-2cbf55e10471 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:19.276272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d74a62f1-982f-468c-bbbe-2cbf55e10471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:19.300284 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74a62f1-982f-468c-bbbe-2cbf55e10471 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:19.310633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bf1807-6a1a-4338-b92a-276c2d04b89f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:19.311728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bf1807-6a1a-4338-b92a-276c2d04b89f admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:19.313308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bf1807-6a1a-4338-b92a-276c2d04b89f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:19.313752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bf1807-6a1a-4338-b92a-276c2d04b89f admin admin] PATH_INFO: `/v3/projects/51548eca8a9844169642e5ae35975b52/users/72c2bc03b4e44dc1b2e1b6ec0f54c4ec/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:19.334745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bf1807-6a1a-4338-b92a-276c2d04b89f admin admin] RBAC: Authorizing `identity:create_grant(project_id=51548eca8a9844169642e5ae35975b52, user_id=72c2bc03b4e44dc1b2e1b6ec0f54c4ec, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:19.343229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bf1807-6a1a-4338-b92a-276c2d04b89f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:19.353341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d10773a1-2788-4fe5-9c6d-5536c70cb902 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d82f770c9534c14b4cec601d5dd68b9', 'request_id': 'req-d10773a1-2788-4fe5-9c6d-5536c70cb902'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.366472 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d10773a1-2788-4fe5-9c6d-5536c70cb902 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d82f770c9534c14b4cec601d5dd68b9', 'request_id': 'req-d10773a1-2788-4fe5-9c6d-5536c70cb902'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.373940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 988/1982] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:19 2022] PUT /identity/v3/projects/51548eca8a9844169642e5ae35975b52/users/72c2bc03b4e44dc1b2e1b6ec0f54c4ec/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:19.376107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bf1807-6a1a-4338-b92a-276c2d04b89f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:19.392537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d10773a1-2788-4fe5-9c6d-5536c70cb902 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d82f770c9534c14b4cec601d5dd68b9', 'request_id': 'req-d10773a1-2788-4fe5-9c6d-5536c70cb902'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.409002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d10773a1-2788-4fe5-9c6d-5536c70cb902 admin admin] The token cache is being invalidate because project 8d82f770c9534c14b4cec601d5dd68b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:19.409337 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d10773a1-2788-4fe5-9c6d-5536c70cb902 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.410317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d10773a1-2788-4fe5-9c6d-5536c70cb902 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.411937 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 995/1983] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:19 2022] DELETE /identity/v3/projects/8d82f770c9534c14b4cec601d5dd68b9 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:19.416786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968e58e4-83be-4136-81f0-f1c31bbe8004 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:19.417105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968e58e4-83be-4136-81f0-f1c31bbe8004 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:19.417354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968e58e4-83be-4136-81f0-f1c31bbe8004 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:19.423773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:19.469569 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-968e58e4-83be-4136-81f0-f1c31bbe8004 None None] MFA Rules not processed for user `72c2bc03b4e44dc1b2e1b6ec0f54c4ec`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:19.473208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-968e58e4-83be-4136-81f0-f1c31bbe8004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:19.509801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bf1807-6a1a-4338-b92a-276c2d04b89f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:19.517549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e95a53-c5e4-4285-9fdc-c5fb0e807025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:19.518776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e95a53-c5e4-4285-9fdc-c5fb0e807025 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:19.519249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e95a53-c5e4-4285-9fdc-c5fb0e807025 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:19.519788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e95a53-c5e4-4285-9fdc-c5fb0e807025 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:19.521025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e95a53-c5e4-4285-9fdc-c5fb0e807025 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:19.521542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e95a53-c5e4-4285-9fdc-c5fb0e807025 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:19.523091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:19.526912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e95a53-c5e4-4285-9fdc-c5fb0e807025 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:19.550339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 989/1984] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:09:19.551962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e95a53-c5e4-4285-9fdc-c5fb0e807025 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:19.560043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c7e95a53-c5e4-4285-9fdc-c5fb0e807025 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:19.576115 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e95a53-c5e4-4285-9fdc-c5fb0e807025 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:19.584478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:19.585147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:19.585704 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:19.586319 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] PATH_INFO: `/v3/users/375882268c5d4366b8411c62e2200aa3` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:19.588710 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] RBAC: Authorizing `identity:delete_user(user_id=375882268c5d4366b8411c62e2200aa3)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:19.592662 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:19.675647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 996/1985] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:19 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:19.675647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968e58e4-83be-4136-81f0-f1c31bbe8004 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:19.675647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-968e58e4-83be-4136-81f0-f1c31bbe8004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:19.683644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '375882268c5d4366b8411c62e2200aa3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.702471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '375882268c5d4366b8411c62e2200aa3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.703935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '375882268c5d4366b8411c62e2200aa3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.706468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968e58e4-83be-4136-81f0-f1c31bbe8004 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:19.715113 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06be317f-350f-46dc-beb7-b22427c7e9dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:19.716217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06be317f-350f-46dc-beb7-b22427c7e9dd admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:19.716679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06be317f-350f-46dc-beb7-b22427c7e9dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:19.717339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06be317f-350f-46dc-beb7-b22427c7e9dd admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:19.718595 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06be317f-350f-46dc-beb7-b22427c7e9dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:19.721950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06be317f-350f-46dc-beb7-b22427c7e9dd admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:19.738589 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '375882268c5d4366b8411c62e2200aa3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.740075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '375882268c5d4366b8411c62e2200aa3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:19.774546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 997/1986] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:19 2022] POST /identity/v3/projects => generated 385 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:19.774546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06be317f-350f-46dc-beb7-b22427c7e9dd admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:19.777294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 990/1987] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:19 2022] DELETE /identity/v3/users/375882268c5d4366b8411c62e2200aa3 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:19.779101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:19.782170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:19.789454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-06be317f-350f-46dc-beb7-b22427c7e9dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:19.988676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c07ea5a-3f3c-4fa6-b00b-b33459b97127 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:19.996596 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06be317f-350f-46dc-beb7-b22427c7e9dd admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:20.002146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0644d072-ca87-4041-b04a-914ec6b5a4bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:20.002599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28358977-ff8b-4c31-a34c-732e3dfdb754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:20.003146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28358977-ff8b-4c31-a34c-732e3dfdb754 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:20.003460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28358977-ff8b-4c31-a34c-732e3dfdb754 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:20.003845 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28358977-ff8b-4c31-a34c-732e3dfdb754 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:20.005330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0644d072-ca87-4041-b04a-914ec6b5a4bf admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:20.005524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28358977-ff8b-4c31-a34c-732e3dfdb754 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:20.005680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0644d072-ca87-4041-b04a-914ec6b5a4bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:20.005680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0644d072-ca87-4041-b04a-914ec6b5a4bf admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:20.006623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28358977-ff8b-4c31-a34c-732e3dfdb754 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:20.020023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0644d072-ca87-4041-b04a-914ec6b5a4bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:20.032063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0644d072-ca87-4041-b04a-914ec6b5a4bf admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:20.075875 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 998/1988] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:19 2022] POST /identity/v3/projects => generated 391 bytes in 305 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:20.081316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28358977-ff8b-4c31-a34c-732e3dfdb754 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:20.086152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 991/1989] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:19 2022] POST /identity/v3/users => generated 293 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:20.092977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0644d072-ca87-4041-b04a-914ec6b5a4bf admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:20.138093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0644d072-ca87-4041-b04a-914ec6b5a4bf admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:20.172588 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:20.210300 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28358977-ff8b-4c31-a34c-732e3dfdb754 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:20.228975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31222411-f15e-46db-a1de-b5b2a61e800f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:20.229662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31222411-f15e-46db-a1de-b5b2a61e800f admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:20.229954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31222411-f15e-46db-a1de-b5b2a61e800f admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:20.231488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31222411-f15e-46db-a1de-b5b2a61e800f admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:20.238272 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31222411-f15e-46db-a1de-b5b2a61e800f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:20.247258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31222411-f15e-46db-a1de-b5b2a61e800f admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:20.327740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 999/1990] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:20 2022] POST /identity/v3/users => generated 302 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:09:20.335666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31222411-f15e-46db-a1de-b5b2a61e800f admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:20.355488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ef845-d5e6-49f9-aaef-6e5ebddb1854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:20.357421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ef845-d5e6-49f9-aaef-6e5ebddb1854 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:20.357917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ef845-d5e6-49f9-aaef-6e5ebddb1854 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:20.358945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ef845-d5e6-49f9-aaef-6e5ebddb1854 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:20.360535 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0ef845-d5e6-49f9-aaef-6e5ebddb1854 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:20.366310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0ef845-d5e6-49f9-aaef-6e5ebddb1854 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:20.381822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31222411-f15e-46db-a1de-b5b2a61e800f admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:20.391806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11402f72-d154-4d2d-982a-20d73b74a27c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:20.392900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11402f72-d154-4d2d-982a-20d73b74a27c admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:20.393244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11402f72-d154-4d2d-982a-20d73b74a27c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:20.393579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11402f72-d154-4d2d-982a-20d73b74a27c admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:20.394374 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11402f72-d154-4d2d-982a-20d73b74a27c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:20.397567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11402f72-d154-4d2d-982a-20d73b74a27c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:20.399916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:20.419604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 992/1991] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:20 2022] GET /identity/v3/roles => generated 1420 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:20.424147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ef845-d5e6-49f9-aaef-6e5ebddb1854 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:20.426565 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1000/1992] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:20 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:20.434818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11402f72-d154-4d2d-982a-20d73b74a27c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:20.454223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ef845-d5e6-49f9-aaef-6e5ebddb1854 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:20.455380 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11402f72-d154-4d2d-982a-20d73b74a27c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:20.464055 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aef02b9-8c88-4bd4-b908-f66899281dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:20.465388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aef02b9-8c88-4bd4-b908-f66899281dc6 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:20.465898 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aef02b9-8c88-4bd4-b908-f66899281dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:20.466398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aef02b9-8c88-4bd4-b908-f66899281dc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:20.467281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ea84a0-a11a-49a7-98b2-af21d1644981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:20.467734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aef02b9-8c88-4bd4-b908-f66899281dc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:20.468234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ea84a0-a11a-49a7-98b2-af21d1644981 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:20.468602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ea84a0-a11a-49a7-98b2-af21d1644981 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:20.468930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ea84a0-a11a-49a7-98b2-af21d1644981 admin admin] PATH_INFO: `/v3/projects/06c48f58c980435cbd9cf2f453149486/users/be7c954dbbfe412a95cefc7e1ff91667/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:20.471235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aef02b9-8c88-4bd4-b908-f66899281dc6 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:20.497343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ea84a0-a11a-49a7-98b2-af21d1644981 admin admin] RBAC: Authorizing `identity:create_grant(project_id=06c48f58c980435cbd9cf2f453149486, user_id=be7c954dbbfe412a95cefc7e1ff91667, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:20.513848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ea84a0-a11a-49a7-98b2-af21d1644981 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:20.539862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 993/1993] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:20 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:20.540830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aef02b9-8c88-4bd4-b908-f66899281dc6 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:20.574181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aef02b9-8c88-4bd4-b908-f66899281dc6 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:20.601789 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1001/1994] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:20 2022] PUT /identity/v3/projects/06c48f58c980435cbd9cf2f453149486/users/be7c954dbbfe412a95cefc7e1ff91667/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:20.611457 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ea84a0-a11a-49a7-98b2-af21d1644981 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:20.771460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7d113c-cca3-4465-ae5c-35df7c3828e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:20.772171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7d113c-cca3-4465-ae5c-35df7c3828e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:20.772737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7d113c-cca3-4465-ae5c-35df7c3828e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:20.773018 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7d113c-cca3-4465-ae5c-35df7c3828e5 admin admin] PATH_INFO: `/v3/projects/27425f0099e345e2bf1d2f182553a8ea/users/8c36228001d548ddb5dbed7ed833cd3f/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:20.798709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7d113c-cca3-4465-ae5c-35df7c3828e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27425f0099e345e2bf1d2f182553a8ea, user_id=8c36228001d548ddb5dbed7ed833cd3f, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:20.802208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7d113c-cca3-4465-ae5c-35df7c3828e5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:20.811780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ea84a0-a11a-49a7-98b2-af21d1644981 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:20.822402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb6a349-0f9f-413f-852a-69bade9dbc41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:20.823561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb6a349-0f9f-413f-852a-69bade9dbc41 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:20.824049 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb6a349-0f9f-413f-852a-69bade9dbc41 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:20.824503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb6a349-0f9f-413f-852a-69bade9dbc41 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:20.825855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb6a349-0f9f-413f-852a-69bade9dbc41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:20.826840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 994/1995] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:20 2022] PUT /identity/v3/projects/27425f0099e345e2bf1d2f182553a8ea/users/8c36228001d548ddb5dbed7ed833cd3f/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:20.827775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7d113c-cca3-4465-ae5c-35df7c3828e5 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:20.830468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1a7d113c-cca3-4465-ae5c-35df7c3828e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:20.842130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb6a349-0f9f-413f-852a-69bade9dbc41 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:20.859176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1002/1996] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:20 2022] GET /identity/v3/roles => generated 1420 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:20.861317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a95c564-c39f-4300-8cff-7691333983a3 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:20.861722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a95c564-c39f-4300-8cff-7691333983a3 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:20.862008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a95c564-c39f-4300-8cff-7691333983a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:20.918552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-5a95c564-c39f-4300-8cff-7691333983a3 None None] MFA Rules not processed for user `8c36228001d548ddb5dbed7ed833cd3f`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:20.921138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5a95c564-c39f-4300-8cff-7691333983a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:20.956378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7d113c-cca3-4465-ae5c-35df7c3828e5 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:20.965776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:20.966662 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:20.967073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:20.967801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] PATH_INFO: `/v3/projects/ade3f2d21b714fdf861390c0c6069758` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:20.970447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ade3f2d21b714fdf861390c0c6069758)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:20.974344 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:21.043700 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=1,keystone:DELETE=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:21.080451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1003/1997] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:20 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:21.081481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a95c564-c39f-4300-8cff-7691333983a3 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:21.095143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a95c564-c39f-4300-8cff-7691333983a3 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:21.104179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ade3f2d21b714fdf861390c0c6069758', 'request_id': 'req-6b91834c-f465-4c79-bb65-054ee1a29b91'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:21.118498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ade3f2d21b714fdf861390c0c6069758', 'request_id': 'req-6b91834c-f465-4c79-bb65-054ee1a29b91'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:21.137070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ade3f2d21b714fdf861390c0c6069758', 'request_id': 'req-6b91834c-f465-4c79-bb65-054ee1a29b91'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:21.162533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] The token cache is being invalidate because project ade3f2d21b714fdf861390c0c6069758 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:21.162982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:21.164394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:21.167581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 995/1998] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:20 2022] DELETE /identity/v3/projects/ade3f2d21b714fdf861390c0c6069758 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:21.168891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:21.172923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:21.208653 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:21.217710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72d3755-8a99-4bea-adfa-e8088a7d4f36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:21.219678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72d3755-8a99-4bea-adfa-e8088a7d4f36 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:21.220517 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72d3755-8a99-4bea-adfa-e8088a7d4f36 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:21.221117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72d3755-8a99-4bea-adfa-e8088a7d4f36 admin admin] PATH_INFO: `/v3/projects/06c48f58c980435cbd9cf2f453149486/users/be7c954dbbfe412a95cefc7e1ff91667/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:21.224067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72d3755-8a99-4bea-adfa-e8088a7d4f36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=06c48f58c980435cbd9cf2f453149486, user_id=be7c954dbbfe412a95cefc7e1ff91667, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:21.228038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72d3755-8a99-4bea-adfa-e8088a7d4f36 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:21.259217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1004/1999] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:21 2022] PUT /identity/v3/projects/06c48f58c980435cbd9cf2f453149486/users/be7c954dbbfe412a95cefc7e1ff91667/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:21.260867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcf6f21-5683-4758-8677-b9e85173e6b7 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:21.261308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcf6f21-5683-4758-8677-b9e85173e6b7 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:21.261586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcf6f21-5683-4758-8677-b9e85173e6b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:21.297519 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-4bcf6f21-5683-4758-8677-b9e85173e6b7 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:21.298925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4bcf6f21-5683-4758-8677-b9e85173e6b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:21.316016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b91834c-f465-4c79-bb65-054ee1a29b91 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:21.410270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1005/2000] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:21.411958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51e9ed7-4cc7-4053-b00a-9718bb3ffe88 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:21.412587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51e9ed7-4cc7-4053-b00a-9718bb3ffe88 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:21.413047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51e9ed7-4cc7-4053-b00a-9718bb3ffe88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:21.431537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f29dfee-c900-4071-96be-0b34eaddaf45 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:21.432174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f29dfee-c900-4071-96be-0b34eaddaf45 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:21.432480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f29dfee-c900-4071-96be-0b34eaddaf45 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:21.432971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f29dfee-c900-4071-96be-0b34eaddaf45 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:21.435585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9f29dfee-c900-4071-96be-0b34eaddaf45 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:21.452922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f51e9ed7-4cc7-4053-b00a-9718bb3ffe88 None None] MFA Rules not processed for user `be7c954dbbfe412a95cefc7e1ff91667`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:21.455468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f51e9ed7-4cc7-4053-b00a-9718bb3ffe88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:21.527387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f29dfee-c900-4071-96be-0b34eaddaf45 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:21.528928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f29dfee-c900-4071-96be-0b34eaddaf45 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:21.541345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 996/2001] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:21 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:21.542721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66044cf-56ae-4126-999b-73d365a7c38a None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:21.543007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66044cf-56ae-4126-999b-73d365a7c38a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:21.543290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66044cf-56ae-4126-999b-73d365a7c38a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:21.566109 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-e66044cf-56ae-4126-999b-73d365a7c38a None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:21.567676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e66044cf-56ae-4126-999b-73d365a7c38a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:21.578672 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 997/2002] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:21.585181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66044cf-56ae-4126-999b-73d365a7c38a None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:21.585887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1006/2003] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:21 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:21.599611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51e9ed7-4cc7-4053-b00a-9718bb3ffe88 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:21.601755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66044cf-56ae-4126-999b-73d365a7c38a None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:21.602107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f51e9ed7-4cc7-4053-b00a-9718bb3ffe88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:21.607703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8eb7b9c-86db-4981-9cda-8ecaee8ad8b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:21.608576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8eb7b9c-86db-4981-9cda-8ecaee8ad8b9 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:21.609029 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8eb7b9c-86db-4981-9cda-8ecaee8ad8b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:21.609480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8eb7b9c-86db-4981-9cda-8ecaee8ad8b9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:21.610359 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8eb7b9c-86db-4981-9cda-8ecaee8ad8b9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:21.611014 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8eb7b9c-86db-4981-9cda-8ecaee8ad8b9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:21.612848 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8eb7b9c-86db-4981-9cda-8ecaee8ad8b9 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:21.623893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f51e9ed7-4cc7-4053-b00a-9718bb3ffe88 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:21.630013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 998/2004] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:09:21.630349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04cc45b-c7ea-4d6c-b595-04989ba30ac8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:21.631668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04cc45b-c7ea-4d6c-b595-04989ba30ac8 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:21.631981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04cc45b-c7ea-4d6c-b595-04989ba30ac8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:21.632386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04cc45b-c7ea-4d6c-b595-04989ba30ac8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:21.634818 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8eb7b9c-86db-4981-9cda-8ecaee8ad8b9 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:21.648137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8eb7b9c-86db-4981-9cda-8ecaee8ad8b9 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:21.653851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39779fc-aacb-4b06-90d8-a58f623ba24c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:21.654490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39779fc-aacb-4b06-90d8-a58f623ba24c admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:21.654771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39779fc-aacb-4b06-90d8-a58f623ba24c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:21.655049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39779fc-aacb-4b06-90d8-a58f623ba24c admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:21.655213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04cc45b-c7ea-4d6c-b595-04989ba30ac8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:21.655789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39779fc-aacb-4b06-90d8-a58f623ba24c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:21.657548 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04cc45b-c7ea-4d6c-b595-04989ba30ac8 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:21.658545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39779fc-aacb-4b06-90d8-a58f623ba24c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:21.665600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:21.674257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1007/2005] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:21 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:09:21.720467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 999/2006] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:21 2022] POST /identity/v3/projects => generated 401 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:21.727542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04cc45b-c7ea-4d6c-b595-04989ba30ac8 service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:21.746102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04cc45b-c7ea-4d6c-b595-04989ba30ac8 service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:21.804428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39779fc-aacb-4b06-90d8-a58f623ba24c admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:21.878722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13eb2e84-10a8-4e49-b67e-fef4ba53faf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:21.879889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13eb2e84-10a8-4e49-b67e-fef4ba53faf5 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:21.880493 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13eb2e84-10a8-4e49-b67e-fef4ba53faf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:21.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13eb2e84-10a8-4e49-b67e-fef4ba53faf5 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:21.884079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13eb2e84-10a8-4e49-b67e-fef4ba53faf5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:21.888189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13eb2e84-10a8-4e49-b67e-fef4ba53faf5 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:21.889103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39779fc-aacb-4b06-90d8-a58f623ba24c admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:21.894577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f9f0b3-0048-4a8f-a6f8-2880c8fb47b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:21.895221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f9f0b3-0048-4a8f-a6f8-2880c8fb47b6 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:21.895559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f9f0b3-0048-4a8f-a6f8-2880c8fb47b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:21.895838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f9f0b3-0048-4a8f-a6f8-2880c8fb47b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:21.898807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-56f9f0b3-0048-4a8f-a6f8-2880c8fb47b6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:21.910842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:21.947168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1008/2007] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:21 2022] POST /identity/v3/users => generated 301 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:21.949204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13eb2e84-10a8-4e49-b67e-fef4ba53faf5 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:21.970385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13eb2e84-10a8-4e49-b67e-fef4ba53faf5 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:21.995704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26d63d5-8fb0-432a-a717-21326c9b99e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:21.995704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26d63d5-8fb0-432a-a717-21326c9b99e8 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:21.995704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26d63d5-8fb0-432a-a717-21326c9b99e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:21.995704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26d63d5-8fb0-432a-a717-21326c9b99e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:21.996437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26d63d5-8fb0-432a-a717-21326c9b99e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:21.996437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26d63d5-8fb0-432a-a717-21326c9b99e8 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:22.029674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1009/2008] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:21 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:22.033861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26d63d5-8fb0-432a-a717-21326c9b99e8 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:22.054389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f9f0b3-0048-4a8f-a6f8-2880c8fb47b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:22.055519 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26d63d5-8fb0-432a-a717-21326c9b99e8 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:22.060476 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f9f0b3-0048-4a8f-a6f8-2880c8fb47b6 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:22.062201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e929a70a-0234-4974-b2ba-881ca23f82fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:22.063773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e929a70a-0234-4974-b2ba-881ca23f82fd admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:22.064128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e929a70a-0234-4974-b2ba-881ca23f82fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:22.064479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e929a70a-0234-4974-b2ba-881ca23f82fd admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:22.065391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e929a70a-0234-4974-b2ba-881ca23f82fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:22.067387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e929a70a-0234-4974-b2ba-881ca23f82fd admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:22.088577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1000/2009] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:21 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:09:22.111280 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f9f0b3-0048-4a8f-a6f8-2880c8fb47b6 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:22.111868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1010/2010] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:22 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:09:22.127994 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f9f0b3-0048-4a8f-a6f8-2880c8fb47b6 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:22.136549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e45f7d2-6b45-4f30-9231-db173078e9dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:22.138483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e45f7d2-6b45-4f30-9231-db173078e9dc admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:22.139207 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e45f7d2-6b45-4f30-9231-db173078e9dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:22.140185 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e45f7d2-6b45-4f30-9231-db173078e9dc admin admin] PATH_INFO: `/v3/projects/543ce218e040418caf8cbb61c3ccf2ca/users/0096c33861944963af89ca88dc096f01/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:22.176807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e45f7d2-6b45-4f30-9231-db173078e9dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=543ce218e040418caf8cbb61c3ccf2ca, user_id=0096c33861944963af89ca88dc096f01, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:22.182394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e45f7d2-6b45-4f30-9231-db173078e9dc admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:22.208996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1001/2011] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:22 2022] PUT /identity/v3/projects/543ce218e040418caf8cbb61c3ccf2ca/users/0096c33861944963af89ca88dc096f01/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:22.230129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc49f7e-8ea4-4e31-b104-e5c4f87ab09a None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:22.230627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc49f7e-8ea4-4e31-b104-e5c4f87ab09a None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:22.231128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc49f7e-8ea4-4e31-b104-e5c4f87ab09a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:22.271475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-8fc49f7e-8ea4-4e31-b104-e5c4f87ab09a None None] MFA Rules not processed for user `0096c33861944963af89ca88dc096f01`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:22.272951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8fc49f7e-8ea4-4e31-b104-e5c4f87ab09a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:22.368633 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1011/2012] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:22 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:09:22.373331 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e45f7d2-6b45-4f30-9231-db173078e9dc admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:22.485342 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e45f7d2-6b45-4f30-9231-db173078e9dc admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:22.494655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da89d098-265f-4a68-9872-8231c736db7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:22.495794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da89d098-265f-4a68-9872-8231c736db7d admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:22.496303 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da89d098-265f-4a68-9872-8231c736db7d admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:22.496807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da89d098-265f-4a68-9872-8231c736db7d admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:22.498112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da89d098-265f-4a68-9872-8231c736db7d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:22.502271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da89d098-265f-4a68-9872-8231c736db7d admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:22.549046 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:22.579414 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1002/2013] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:22 2022] POST /identity/v3/projects => generated 399 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:22.582882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc49f7e-8ea4-4e31-b104-e5c4f87ab09a None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:22.600965 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc49f7e-8ea4-4e31-b104-e5c4f87ab09a None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:22.674198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ee671a-f4ec-45fc-8c88-1920b92a7ae3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:22.674879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ee671a-f4ec-45fc-8c88-1920b92a7ae3 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:22.675174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ee671a-f4ec-45fc-8c88-1920b92a7ae3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:22.675491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ee671a-f4ec-45fc-8c88-1920b92a7ae3 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:22.676162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ee671a-f4ec-45fc-8c88-1920b92a7ae3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:22.678220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ee671a-f4ec-45fc-8c88-1920b92a7ae3 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:22.703344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1012/2014] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:22 2022] POST /identity/v3/users => generated 306 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:09:22.709081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da89d098-265f-4a68-9872-8231c736db7d admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:22.728347 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da89d098-265f-4a68-9872-8231c736db7d admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:22.737645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5db7d3f-3a41-497a-ae31-3fd783affbb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:22.738385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5db7d3f-3a41-497a-ae31-3fd783affbb4 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:22.738892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5db7d3f-3a41-497a-ae31-3fd783affbb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:22.739863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5db7d3f-3a41-497a-ae31-3fd783affbb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:22.740645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5db7d3f-3a41-497a-ae31-3fd783affbb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:22.742472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5db7d3f-3a41-497a-ae31-3fd783affbb4 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:22.773666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1003/2015] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:22 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:22.775020 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ee671a-f4ec-45fc-8c88-1920b92a7ae3 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:22.787008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ee671a-f4ec-45fc-8c88-1920b92a7ae3 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:22.792072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bb6c81-c156-43ab-82f1-41b3ba03a172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:22.792785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bb6c81-c156-43ab-82f1-41b3ba03a172 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:22.793062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bb6c81-c156-43ab-82f1-41b3ba03a172 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:22.793348 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bb6c81-c156-43ab-82f1-41b3ba03a172 admin admin] PATH_INFO: `/v3/projects/5ffdec31c9124a948f33676f99c69c39/users/bff91cdb575f4c36bafcf2fc7ae1f5a0/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:22.821829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3bb6c81-c156-43ab-82f1-41b3ba03a172 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ffdec31c9124a948f33676f99c69c39, user_id=bff91cdb575f4c36bafcf2fc7ae1f5a0, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:22.827184 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3bb6c81-c156-43ab-82f1-41b3ba03a172 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:22.856693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1013/2016] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:22 2022] PUT /identity/v3/projects/5ffdec31c9124a948f33676f99c69c39/users/bff91cdb575f4c36bafcf2fc7ae1f5a0/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:22.862277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5db7d3f-3a41-497a-ae31-3fd783affbb4 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:22.890268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5db7d3f-3a41-497a-ae31-3fd783affbb4 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:22.961789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85f44c8-7004-4e5a-acae-cc760ed69c99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:22.962448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85f44c8-7004-4e5a-acae-cc760ed69c99 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:22.962731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85f44c8-7004-4e5a-acae-cc760ed69c99 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:22.963009 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85f44c8-7004-4e5a-acae-cc760ed69c99 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:22.963736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85f44c8-7004-4e5a-acae-cc760ed69c99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:22.965359 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85f44c8-7004-4e5a-acae-cc760ed69c99 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:22.976630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1004/2017] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:22 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:22.981568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bb6c81-c156-43ab-82f1-41b3ba03a172 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:22.992955 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bb6c81-c156-43ab-82f1-41b3ba03a172 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:22.997830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b705cc-ffa9-4fd9-8595-c67742253e4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:22.998535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b705cc-ffa9-4fd9-8595-c67742253e4d admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:22.998815 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b705cc-ffa9-4fd9-8595-c67742253e4d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:22.999095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b705cc-ffa9-4fd9-8595-c67742253e4d admin admin] PATH_INFO: `/v3/projects/5ffdec31c9124a948f33676f99c69c39/users/bff91cdb575f4c36bafcf2fc7ae1f5a0/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:23.001187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b705cc-ffa9-4fd9-8595-c67742253e4d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ffdec31c9124a948f33676f99c69c39, user_id=bff91cdb575f4c36bafcf2fc7ae1f5a0, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:23.003912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b705cc-ffa9-4fd9-8595-c67742253e4d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:23.016410 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1014/2018] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:22 2022] PUT /identity/v3/projects/5ffdec31c9124a948f33676f99c69c39/users/bff91cdb575f4c36bafcf2fc7ae1f5a0/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:23.026969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0662464a-0a11-4774-9436-5172e4e3a5bc None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:23.027253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0662464a-0a11-4774-9436-5172e4e3a5bc None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:23.027564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0662464a-0a11-4774-9436-5172e4e3a5bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:23.058683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-0662464a-0a11-4774-9436-5172e4e3a5bc None None] MFA Rules not processed for user `bff91cdb575f4c36bafcf2fc7ae1f5a0`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:23.060885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0662464a-0a11-4774-9436-5172e4e3a5bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:23.152182 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1005/2019] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:23 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:23.164468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b705cc-ffa9-4fd9-8595-c67742253e4d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:23.183026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:23.232247 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b705cc-ffa9-4fd9-8595-c67742253e4d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:23.236407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d066616-2513-4f4d-be83-a040d4fa0acd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:23.237183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d066616-2513-4f4d-be83-a040d4fa0acd service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:23.237432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d066616-2513-4f4d-be83-a040d4fa0acd service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:23.237773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d066616-2513-4f4d-be83-a040d4fa0acd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:23.248816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d066616-2513-4f4d-be83-a040d4fa0acd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:23.250856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d066616-2513-4f4d-be83-a040d4fa0acd service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:23.261842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1015/2020] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:23 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:09:23.345643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0662464a-0a11-4774-9436-5172e4e3a5bc None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:23.355274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0662464a-0a11-4774-9436-5172e4e3a5bc None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:23.358936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d826b417-6d62-46d1-a7ee-b22d72420007 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:23.359653 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d826b417-6d62-46d1-a7ee-b22d72420007 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:23.359927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d826b417-6d62-46d1-a7ee-b22d72420007 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:23.360193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d826b417-6d62-46d1-a7ee-b22d72420007 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:23.432859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d826b417-6d62-46d1-a7ee-b22d72420007 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:23.435903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d826b417-6d62-46d1-a7ee-b22d72420007 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:23.454286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1006/2021] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:23 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:09:33.256928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=20 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:33.445561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=96 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:34.383643 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d066616-2513-4f4d-be83-a040d4fa0acd service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:34.471172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d066616-2513-4f4d-be83-a040d4fa0acd service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:34.476638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:34.477301 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:34.477584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:34.477862 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] PATH_INFO: `/v3/users/0096c33861944963af89ca88dc096f01` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:34.479142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] RBAC: Authorizing `identity:delete_user(user_id=0096c33861944963af89ca88dc096f01)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:34.481177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:34.559845 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0096c33861944963af89ca88dc096f01'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:34.560849 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0096c33861944963af89ca88dc096f01'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:34.572135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0096c33861944963af89ca88dc096f01'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:34.597862 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0096c33861944963af89ca88dc096f01'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:34.606461 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0096c33861944963af89ca88dc096f01'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:34.619196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1016/2022] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:34 2022] DELETE /identity/v3/users/0096c33861944963af89ca88dc096f01 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:34.626001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4486e1-77e0-4a84-b74a-d546599b042b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:34.626802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4486e1-77e0-4a84-b74a-d546599b042b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:34.626802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4486e1-77e0-4a84-b74a-d546599b042b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:34.651177 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ab4486e1-77e0-4a84-b74a-d546599b042b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:34.652918 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ab4486e1-77e0-4a84-b74a-d546599b042b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:34.727550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1007/2023] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:34.741251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:34.743945 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:34.832546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000120d2-2074-4ce4-9cf7-929fb8bc8cbb admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:34.836743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e88cef0-5605-4a1d-8a41-d89b1c6e7b21 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:34.837390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e88cef0-5605-4a1d-8a41-d89b1c6e7b21 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:34.837667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e88cef0-5605-4a1d-8a41-d89b1c6e7b21 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:34.837942 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e88cef0-5605-4a1d-8a41-d89b1c6e7b21 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:34.848388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e88cef0-5605-4a1d-8a41-d89b1c6e7b21 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:34.849868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e88cef0-5605-4a1d-8a41-d89b1c6e7b21 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:34.861990 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1017/2024] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:35.208551 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4486e1-77e0-4a84-b74a-d546599b042b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:35.210234 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ab4486e1-77e0-4a84-b74a-d546599b042b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:35.228656 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4486e1-77e0-4a84-b74a-d546599b042b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:35.237623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20931c0-d3cd-4d37-9dbc-9f4ff7777c51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:35.238355 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20931c0-d3cd-4d37-9dbc-9f4ff7777c51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:35.238713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20931c0-d3cd-4d37-9dbc-9f4ff7777c51 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:35.239002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20931c0-d3cd-4d37-9dbc-9f4ff7777c51 admin admin] PATH_INFO: `/v3/users/72c2bc03b4e44dc1b2e1b6ec0f54c4ec` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:35.240716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20931c0-d3cd-4d37-9dbc-9f4ff7777c51 admin admin] RBAC: Authorizing `identity:delete_user(user_id=72c2bc03b4e44dc1b2e1b6ec0f54c4ec)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:35.243020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20931c0-d3cd-4d37-9dbc-9f4ff7777c51 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:35.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a20931c0-d3cd-4d37-9dbc-9f4ff7777c51 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '72c2bc03b4e44dc1b2e1b6ec0f54c4ec'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.333870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a20931c0-d3cd-4d37-9dbc-9f4ff7777c51 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '72c2bc03b4e44dc1b2e1b6ec0f54c4ec'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.334713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a20931c0-d3cd-4d37-9dbc-9f4ff7777c51 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '72c2bc03b4e44dc1b2e1b6ec0f54c4ec'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.345794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e88cef0-5605-4a1d-8a41-d89b1c6e7b21 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:35.347465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-1e88cef0-5605-4a1d-8a41-d89b1c6e7b21 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:35.375399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e88cef0-5605-4a1d-8a41-d89b1c6e7b21 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:35.383655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:35.384360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:35.384861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:35.385295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] PATH_INFO: `/v3/projects/543ce218e040418caf8cbb61c3ccf2ca` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:35.386924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] RBAC: Authorizing `identity:delete_project(project_id=543ce218e040418caf8cbb61c3ccf2ca)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:35.390008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:35.404955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a20931c0-d3cd-4d37-9dbc-9f4ff7777c51 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '72c2bc03b4e44dc1b2e1b6ec0f54c4ec'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.407569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a20931c0-d3cd-4d37-9dbc-9f4ff7777c51 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '72c2bc03b4e44dc1b2e1b6ec0f54c4ec'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.415205 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:35.454407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1008/2025] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:35 2022] DELETE /identity/v3/users/72c2bc03b4e44dc1b2e1b6ec0f54c4ec => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:35.460425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0386bb17-7aef-4270-b577-03a2d69d2a95 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:35.461459 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0386bb17-7aef-4270-b577-03a2d69d2a95 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:35.462407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0386bb17-7aef-4270-b577-03a2d69d2a95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:35.470634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '543ce218e040418caf8cbb61c3ccf2ca', 'request_id': 'req-0330fb7b-8f2e-403f-856d-3856cd79d590'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.479185 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '543ce218e040418caf8cbb61c3ccf2ca', 'request_id': 'req-0330fb7b-8f2e-403f-856d-3856cd79d590'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.491233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-0386bb17-7aef-4270-b577-03a2d69d2a95 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:35.492673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '543ce218e040418caf8cbb61c3ccf2ca', 'request_id': 'req-0330fb7b-8f2e-403f-856d-3856cd79d590'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.493263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0386bb17-7aef-4270-b577-03a2d69d2a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:35.516795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] The token cache is being invalidate because project 543ce218e040418caf8cbb61c3ccf2ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:35.517864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.519403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.522390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1018/2026] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:35 2022] DELETE /identity/v3/projects/543ce218e040418caf8cbb61c3ccf2ca => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:35.530320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:35.535555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:35.624976 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1009/2027] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:09:35.637315 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0386bb17-7aef-4270-b577-03a2d69d2a95 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:35.640809 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0386bb17-7aef-4270-b577-03a2d69d2a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:35.665102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:35.670447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0330fb7b-8f2e-403f-856d-3856cd79d590 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:35.682355 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:35.684409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:35.685276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:35.685895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] PATH_INFO: `/v3/users/bff91cdb575f4c36bafcf2fc7ae1f5a0` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:35.689168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bff91cdb575f4c36bafcf2fc7ae1f5a0)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:35.693369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:35.774781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0386bb17-7aef-4270-b577-03a2d69d2a95 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:35.801861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2b81cc-bdae-47e1-8720-65ae458a2b94 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:35.802702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2b81cc-bdae-47e1-8720-65ae458a2b94 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:35.802981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2b81cc-bdae-47e1-8720-65ae458a2b94 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:35.803254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2b81cc-bdae-47e1-8720-65ae458a2b94 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:35.804237 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bff91cdb575f4c36bafcf2fc7ae1f5a0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.805094 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bff91cdb575f4c36bafcf2fc7ae1f5a0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.817684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bff91cdb575f4c36bafcf2fc7ae1f5a0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.818397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2b81cc-bdae-47e1-8720-65ae458a2b94 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:35.820121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2b81cc-bdae-47e1-8720-65ae458a2b94 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:35.835179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1010/2028] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:35.837496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bff91cdb575f4c36bafcf2fc7ae1f5a0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.852554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bff91cdb575f4c36bafcf2fc7ae1f5a0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:35.862644 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1019/2029] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:35 2022] DELETE /identity/v3/users/bff91cdb575f4c36bafcf2fc7ae1f5a0 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:36.177003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2b81cc-bdae-47e1-8720-65ae458a2b94 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:36.178658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-da2b81cc-bdae-47e1-8720-65ae458a2b94 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:36.216054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2b81cc-bdae-47e1-8720-65ae458a2b94 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:36.326666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:36.327702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:36.328272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:36.328625 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] PATH_INFO: `/v3/projects/51548eca8a9844169642e5ae35975b52` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:36.330208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] RBAC: Authorizing `identity:delete_project(project_id=51548eca8a9844169642e5ae35975b52)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:36.332390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:36.341396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:36.343222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:36.371967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b920a66e-b576-4c14-8843-824c0656a632 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:36.394343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:36.395340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:36.395879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:36.396227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] PATH_INFO: `/v3/projects/5ffdec31c9124a948f33676f99c69c39` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:36.399935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ffdec31c9124a948f33676f99c69c39)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:36.403918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:36.406443 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '51548eca8a9844169642e5ae35975b52', 'request_id': 'req-5048b640-fde0-4bed-9f71-e665ba4b339f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.413607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '51548eca8a9844169642e5ae35975b52', 'request_id': 'req-5048b640-fde0-4bed-9f71-e665ba4b339f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.424573 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '51548eca8a9844169642e5ae35975b52', 'request_id': 'req-5048b640-fde0-4bed-9f71-e665ba4b339f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.435671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:36.447925 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] The token cache is being invalidate because project 51548eca8a9844169642e5ae35975b52 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:36.448388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.449499 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.459165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1011/2030] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:36 2022] DELETE /identity/v3/projects/51548eca8a9844169642e5ae35975b52 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:36.461657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:36.472195 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:36.502876 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ffdec31c9124a948f33676f99c69c39', 'request_id': 'req-065eb712-ffbc-44f3-bd53-23684ecf9318'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.518344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ffdec31c9124a948f33676f99c69c39', 'request_id': 'req-065eb712-ffbc-44f3-bd53-23684ecf9318'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.534568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ffdec31c9124a948f33676f99c69c39', 'request_id': 'req-065eb712-ffbc-44f3-bd53-23684ecf9318'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.550633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:36.565657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] The token cache is being invalidate because project 5ffdec31c9124a948f33676f99c69c39 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:36.566180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.567842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.570909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1020/2031] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:36 2022] DELETE /identity/v3/projects/5ffdec31c9124a948f33676f99c69c39 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:36.572265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:36.574728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:36.578566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5048b640-fde0-4bed-9f71-e665ba4b339f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:36.652425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc9f46b-112f-4d30-a568-e0c7525051f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:36.653693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc9f46b-112f-4d30-a568-e0c7525051f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:36.654050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc9f46b-112f-4d30-a568-e0c7525051f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:36.654336 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc9f46b-112f-4d30-a568-e0c7525051f5 admin admin] PATH_INFO: `/v3/users/be7c954dbbfe412a95cefc7e1ff91667` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:36.656873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc9f46b-112f-4d30-a568-e0c7525051f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be7c954dbbfe412a95cefc7e1ff91667)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:36.660914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc9f46b-112f-4d30-a568-e0c7525051f5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:36.665647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065eb712-ffbc-44f3-bd53-23684ecf9318 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:36.670933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644bf6fb-fc5d-480d-b36d-1427230834c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:36.672201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644bf6fb-fc5d-480d-b36d-1427230834c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:36.672516 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644bf6fb-fc5d-480d-b36d-1427230834c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:36.672795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644bf6fb-fc5d-480d-b36d-1427230834c8 admin admin] PATH_INFO: `/v3/users/8c36228001d548ddb5dbed7ed833cd3f` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:36.674076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644bf6fb-fc5d-480d-b36d-1427230834c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c36228001d548ddb5dbed7ed833cd3f)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:36.676149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644bf6fb-fc5d-480d-b36d-1427230834c8 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:36.736194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ebc9f46b-112f-4d30-a568-e0c7525051f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be7c954dbbfe412a95cefc7e1ff91667'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.736479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-644bf6fb-fc5d-480d-b36d-1427230834c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c36228001d548ddb5dbed7ed833cd3f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.736882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-644bf6fb-fc5d-480d-b36d-1427230834c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c36228001d548ddb5dbed7ed833cd3f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.744378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ebc9f46b-112f-4d30-a568-e0c7525051f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be7c954dbbfe412a95cefc7e1ff91667'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.745123 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ebc9f46b-112f-4d30-a568-e0c7525051f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be7c954dbbfe412a95cefc7e1ff91667'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.747090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-644bf6fb-fc5d-480d-b36d-1427230834c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c36228001d548ddb5dbed7ed833cd3f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.762018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-644bf6fb-fc5d-480d-b36d-1427230834c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c36228001d548ddb5dbed7ed833cd3f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.764183 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ebc9f46b-112f-4d30-a568-e0c7525051f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be7c954dbbfe412a95cefc7e1ff91667'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.765444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ebc9f46b-112f-4d30-a568-e0c7525051f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be7c954dbbfe412a95cefc7e1ff91667'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.773783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-644bf6fb-fc5d-480d-b36d-1427230834c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c36228001d548ddb5dbed7ed833cd3f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:36.782679 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1021/2032] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:36 2022] DELETE /identity/v3/users/8c36228001d548ddb5dbed7ed833cd3f => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:36.784228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d53542-885e-4aef-9a3b-9cec5ab79a0f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:36.784638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d53542-885e-4aef-9a3b-9cec5ab79a0f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:36.785009 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d53542-885e-4aef-9a3b-9cec5ab79a0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:36.797315 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1012/2033] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:36 2022] DELETE /identity/v3/users/be7c954dbbfe412a95cefc7e1ff91667 => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:36.799848 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665e4cbb-94d6-4aae-aebd-585319bf2f03 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:36.800449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665e4cbb-94d6-4aae-aebd-585319bf2f03 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:36.801499 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665e4cbb-94d6-4aae-aebd-585319bf2f03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:36.811273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-21d53542-885e-4aef-9a3b-9cec5ab79a0f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:36.812943 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-21d53542-885e-4aef-9a3b-9cec5ab79a0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:36.839400 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-665e4cbb-94d6-4aae-aebd-585319bf2f03 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:36.842053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-665e4cbb-94d6-4aae-aebd-585319bf2f03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:36.852875 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:36.955573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1022/2034] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:36.963203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e599de4-1859-444b-a32f-a7b244927dda None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:36.963662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e599de4-1859-444b-a32f-a7b244927dda None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:36.964041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e599de4-1859-444b-a32f-a7b244927dda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:37.003587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-9e599de4-1859-444b-a32f-a7b244927dda None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:37.011205 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9e599de4-1859-444b-a32f-a7b244927dda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:37.016306 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1013/2035] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:37.032050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1023/2036] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:37.036267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665e4cbb-94d6-4aae-aebd-585319bf2f03 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:37.036913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e599de4-1859-444b-a32f-a7b244927dda None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:37.043268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-665e4cbb-94d6-4aae-aebd-585319bf2f03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:37.056315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e599de4-1859-444b-a32f-a7b244927dda None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:37.066153 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb12d54c-27a7-4c1a-bb66-69da1fca7f32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:37.067076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb12d54c-27a7-4c1a-bb66-69da1fca7f32 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:37.067511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb12d54c-27a7-4c1a-bb66-69da1fca7f32 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:37.067899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb12d54c-27a7-4c1a-bb66-69da1fca7f32 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:37.070085 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb12d54c-27a7-4c1a-bb66-69da1fca7f32 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:37.070085 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb12d54c-27a7-4c1a-bb66-69da1fca7f32 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:37.072628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb12d54c-27a7-4c1a-bb66-69da1fca7f32 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:37.081914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:37.104886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1024/2037] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:09:37.111313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb12d54c-27a7-4c1a-bb66-69da1fca7f32 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:37.131135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb12d54c-27a7-4c1a-bb66-69da1fca7f32 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:37.137136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c435e49-93f7-482f-a01d-a404531193f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:37.138167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c435e49-93f7-482f-a01d-a404531193f8 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:37.139524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c435e49-93f7-482f-a01d-a404531193f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:37.140618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c435e49-93f7-482f-a01d-a404531193f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:37.141690 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c435e49-93f7-482f-a01d-a404531193f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:37.144117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c435e49-93f7-482f-a01d-a404531193f8 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:37.182643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665e4cbb-94d6-4aae-aebd-585319bf2f03 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:37.191966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380ce161-3134-4de5-b3c1-a9980cc6c775 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:37.192722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1025/2038] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:37 2022] POST /identity/v3/projects => generated 389 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:37.193156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380ce161-3134-4de5-b3c1-a9980cc6c775 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:37.193991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380ce161-3134-4de5-b3c1-a9980cc6c775 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:37.194264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c435e49-93f7-482f-a01d-a404531193f8 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:37.195099 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380ce161-3134-4de5-b3c1-a9980cc6c775 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:37.216793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380ce161-3134-4de5-b3c1-a9980cc6c775 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:37.220630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380ce161-3134-4de5-b3c1-a9980cc6c775 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:37.264821 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1014/2039] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:37.284379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380ce161-3134-4de5-b3c1-a9980cc6c775 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:37.288089 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-380ce161-3134-4de5-b3c1-a9980cc6c775 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:37.320151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380ce161-3134-4de5-b3c1-a9980cc6c775 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:37.415077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c435e49-93f7-482f-a01d-a404531193f8 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:37.434499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8281beec-ea42-4e3a-8821-d9f1906e1e57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:37.439460 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8281beec-ea42-4e3a-8821-d9f1906e1e57 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:37.439820 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8281beec-ea42-4e3a-8821-d9f1906e1e57 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:37.440106 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8281beec-ea42-4e3a-8821-d9f1906e1e57 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:37.443616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8281beec-ea42-4e3a-8821-d9f1906e1e57 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:37.448873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8281beec-ea42-4e3a-8821-d9f1906e1e57 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:37.492237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dcc246-44af-4539-a979-73a7f5e738a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:37.492554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1026/2040] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:37 2022] POST /identity/v3/users => generated 295 bytes in 299 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:37.496664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8281beec-ea42-4e3a-8821-d9f1906e1e57 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:37.504968 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dcc246-44af-4539-a979-73a7f5e738a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:37.505777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dcc246-44af-4539-a979-73a7f5e738a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:37.507575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dcc246-44af-4539-a979-73a7f5e738a8 admin admin] PATH_INFO: `/v3/projects/06c48f58c980435cbd9cf2f453149486` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:37.511317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dcc246-44af-4539-a979-73a7f5e738a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=06c48f58c980435cbd9cf2f453149486)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:37.513817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8281beec-ea42-4e3a-8821-d9f1906e1e57 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:37.517618 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dcc246-44af-4539-a979-73a7f5e738a8 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:37.526809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc5c234-a926-4c60-9d49-7231424a16af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:37.529868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc5c234-a926-4c60-9d49-7231424a16af admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:37.531862 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc5c234-a926-4c60-9d49-7231424a16af admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:37.535206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc5c234-a926-4c60-9d49-7231424a16af admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:37.536924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc5c234-a926-4c60-9d49-7231424a16af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:37.548339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc5c234-a926-4c60-9d49-7231424a16af admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:37.610389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1027/2041] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:37 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:37.610389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc5c234-a926-4c60-9d49-7231424a16af admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:37.645200 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-59dcc246-44af-4539-a979-73a7f5e738a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '06c48f58c980435cbd9cf2f453149486', 'request_id': 'req-59dcc246-44af-4539-a979-73a7f5e738a8'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:37.656026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-59dcc246-44af-4539-a979-73a7f5e738a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '06c48f58c980435cbd9cf2f453149486', 'request_id': 'req-59dcc246-44af-4539-a979-73a7f5e738a8'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:37.664563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc5c234-a926-4c60-9d49-7231424a16af admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:37.670171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-59dcc246-44af-4539-a979-73a7f5e738a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '06c48f58c980435cbd9cf2f453149486', 'request_id': 'req-59dcc246-44af-4539-a979-73a7f5e738a8'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:37.692671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-59dcc246-44af-4539-a979-73a7f5e738a8 admin admin] The token cache is being invalidate because project 06c48f58c980435cbd9cf2f453149486 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:37.692947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-59dcc246-44af-4539-a979-73a7f5e738a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:37.695191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-59dcc246-44af-4539-a979-73a7f5e738a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:37.705837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1015/2042] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:37 2022] DELETE /identity/v3/projects/06c48f58c980435cbd9cf2f453149486 => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:37.708079 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b586d2-f2de-4708-a58a-6b9e67fc5982 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:37.708712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b586d2-f2de-4708-a58a-6b9e67fc5982 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:37.708712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b586d2-f2de-4708-a58a-6b9e67fc5982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:37.759727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-08b586d2-f2de-4708-a58a-6b9e67fc5982 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:37.759727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-08b586d2-f2de-4708-a58a-6b9e67fc5982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:37.862535 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:37.882475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161e239c-e267-43cd-8620-adade4f7867a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:37.883869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161e239c-e267-43cd-8620-adade4f7867a admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:37.884215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161e239c-e267-43cd-8620-adade4f7867a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:37.884495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161e239c-e267-43cd-8620-adade4f7867a admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:37.885936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161e239c-e267-43cd-8620-adade4f7867a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:37.888096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161e239c-e267-43cd-8620-adade4f7867a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:37.898088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:37.905512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1028/2043] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:37 2022] GET /identity/v3/roles => generated 1420 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:37.907641 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161e239c-e267-43cd-8620-adade4f7867a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:37.914526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1016/2044] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:37.916162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b586d2-f2de-4708-a58a-6b9e67fc5982 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:37.918748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-08b586d2-f2de-4708-a58a-6b9e67fc5982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:37.930343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-161e239c-e267-43cd-8620-adade4f7867a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:37.944175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b586d2-f2de-4708-a58a-6b9e67fc5982 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:37.950175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba7387d-3229-40ef-b6d7-5a03a8ccf9e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:37.951088 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba7387d-3229-40ef-b6d7-5a03a8ccf9e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:37.951457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba7387d-3229-40ef-b6d7-5a03a8ccf9e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:37.951869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba7387d-3229-40ef-b6d7-5a03a8ccf9e4 admin admin] PATH_INFO: `/v3/projects/5f426702722f491594bb883e4bcd6776/users/e448dc9944a243cf931f63514060ada3/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:37.958860 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161e239c-e267-43cd-8620-adade4f7867a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:37.967775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba79db36-8afe-4f65-a5f8-052daa983c48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:37.969088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba79db36-8afe-4f65-a5f8-052daa983c48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:37.969469 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba79db36-8afe-4f65-a5f8-052daa983c48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:37.969900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba79db36-8afe-4f65-a5f8-052daa983c48 admin admin] PATH_INFO: `/v3/projects/27425f0099e345e2bf1d2f182553a8ea` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:37.971416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba79db36-8afe-4f65-a5f8-052daa983c48 admin admin] RBAC: Authorizing `identity:delete_project(project_id=27425f0099e345e2bf1d2f182553a8ea)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:37.973621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba79db36-8afe-4f65-a5f8-052daa983c48 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:37.980422 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba7387d-3229-40ef-b6d7-5a03a8ccf9e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f426702722f491594bb883e4bcd6776, user_id=e448dc9944a243cf931f63514060ada3, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:37.990147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba7387d-3229-40ef-b6d7-5a03a8ccf9e4 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.017829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1017/2045] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:37 2022] PUT /identity/v3/projects/5f426702722f491594bb883e4bcd6776/users/e448dc9944a243cf931f63514060ada3/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:38.021180 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50b147e-1410-45d2-96da-9f8109005c76 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.021574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50b147e-1410-45d2-96da-9f8109005c76 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.021934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50b147e-1410-45d2-96da-9f8109005c76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.046556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d50b147e-1410-45d2-96da-9f8109005c76 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:38.047895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d50b147e-1410-45d2-96da-9f8109005c76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:38.057751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ba79db36-8afe-4f65-a5f8-052daa983c48 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27425f0099e345e2bf1d2f182553a8ea', 'request_id': 'req-ba79db36-8afe-4f65-a5f8-052daa983c48'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:38.073216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ba79db36-8afe-4f65-a5f8-052daa983c48 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27425f0099e345e2bf1d2f182553a8ea', 'request_id': 'req-ba79db36-8afe-4f65-a5f8-052daa983c48'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:38.085072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ba79db36-8afe-4f65-a5f8-052daa983c48 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27425f0099e345e2bf1d2f182553a8ea', 'request_id': 'req-ba79db36-8afe-4f65-a5f8-052daa983c48'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:38.099832 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ba79db36-8afe-4f65-a5f8-052daa983c48 admin admin] The token cache is being invalidate because project 27425f0099e345e2bf1d2f182553a8ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:38.100200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ba79db36-8afe-4f65-a5f8-052daa983c48 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:38.101814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ba79db36-8afe-4f65-a5f8-052daa983c48 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:38.110778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1029/2046] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:37 2022] DELETE /identity/v3/projects/27425f0099e345e2bf1d2f182553a8ea => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:38.113817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb7d49b-d99a-4373-b676-5f583ade920b None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.114630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb7d49b-d99a-4373-b676-5f583ade920b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.115451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb7d49b-d99a-4373-b676-5f583ade920b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.145158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1018/2047] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:38.147696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba44544d-8e93-4155-8ea2-66d43bd52a6e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.149330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba44544d-8e93-4155-8ea2-66d43bd52a6e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.150737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba44544d-8e93-4155-8ea2-66d43bd52a6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.157097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ddb7d49b-d99a-4373-b676-5f583ade920b None None] MFA Rules not processed for user `e448dc9944a243cf931f63514060ada3`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:38.158447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ddb7d49b-d99a-4373-b676-5f583ade920b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:38.203336 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ba44544d-8e93-4155-8ea2-66d43bd52a6e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:38.206730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ba44544d-8e93-4155-8ea2-66d43bd52a6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:38.229607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1019/2048] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:38.229607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba44544d-8e93-4155-8ea2-66d43bd52a6e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.244537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba44544d-8e93-4155-8ea2-66d43bd52a6e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:38.254156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ed5029-4a8c-411d-a64a-285ccda883b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:38.254891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed5029-4a8c-411d-a64a-285ccda883b9 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.255550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed5029-4a8c-411d-a64a-285ccda883b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.255550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed5029-4a8c-411d-a64a-285ccda883b9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.256132 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ed5029-4a8c-411d-a64a-285ccda883b9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:38.256699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ed5029-4a8c-411d-a64a-285ccda883b9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:38.260914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ed5029-4a8c-411d-a64a-285ccda883b9 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.289498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1020/2049] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:09:38.291956 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4250f78d-2665-4eec-b4e8-5cb5970e3086 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.292168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4250f78d-2665-4eec-b4e8-5cb5970e3086 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.292517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4250f78d-2665-4eec-b4e8-5cb5970e3086 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.304434 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1030/2050] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:38 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:38.307050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb7d49b-d99a-4373-b676-5f583ade920b None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.336885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-4250f78d-2665-4eec-b4e8-5cb5970e3086 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:38.336999 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb7d49b-d99a-4373-b676-5f583ade920b None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:38.338879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-4250f78d-2665-4eec-b4e8-5cb5970e3086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:38.346048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152e6a2d-9764-49ae-ad9d-8e90abc4ca60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:38.347802 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e6a2d-9764-49ae-ad9d-8e90abc4ca60 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.348349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e6a2d-9764-49ae-ad9d-8e90abc4ca60 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.348879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e6a2d-9764-49ae-ad9d-8e90abc4ca60 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.350112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152e6a2d-9764-49ae-ad9d-8e90abc4ca60 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:38.353209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1021/2051] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:38.354400 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4250f78d-2665-4eec-b4e8-5cb5970e3086 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.354527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152e6a2d-9764-49ae-ad9d-8e90abc4ca60 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.356585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-4250f78d-2665-4eec-b4e8-5cb5970e3086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:38.373072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4250f78d-2665-4eec-b4e8-5cb5970e3086 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:38.380698 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35bfa8b-9063-4980-ad72-3884528c654f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:38.382644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35bfa8b-9063-4980-ad72-3884528c654f admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.382931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35bfa8b-9063-4980-ad72-3884528c654f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.383203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35bfa8b-9063-4980-ad72-3884528c654f admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.383924 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35bfa8b-9063-4980-ad72-3884528c654f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:38.385887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35bfa8b-9063-4980-ad72-3884528c654f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.410386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1031/2052] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:38 2022] POST /identity/v3/projects => generated 383 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:38.423037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152e6a2d-9764-49ae-ad9d-8e90abc4ca60 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.428412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1022/2053] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:38 2022] POST /identity/v3/projects => generated 387 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:38.429351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35bfa8b-9063-4980-ad72-3884528c654f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.489908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:38.511275 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152e6a2d-9764-49ae-ad9d-8e90abc4ca60 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:38.548283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35bfa8b-9063-4980-ad72-3884528c654f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:38.557711 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9245aa-7325-444d-9784-c1bc6c1308b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:38.558403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9245aa-7325-444d-9784-c1bc6c1308b0 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.558685 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9245aa-7325-444d-9784-c1bc6c1308b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.559222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9245aa-7325-444d-9784-c1bc6c1308b0 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.559939 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9245aa-7325-444d-9784-c1bc6c1308b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:38.562777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9245aa-7325-444d-9784-c1bc6c1308b0 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.589892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1023/2054] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:38 2022] POST /identity/v3/users => generated 292 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:38.590950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9245aa-7325-444d-9784-c1bc6c1308b0 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.598964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:38.603702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9245aa-7325-444d-9784-c1bc6c1308b0 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:38.611661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65d18d4-34fd-4231-9aa4-8033e414045c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:38.612298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65d18d4-34fd-4231-9aa4-8033e414045c admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.612582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65d18d4-34fd-4231-9aa4-8033e414045c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.612878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65d18d4-34fd-4231-9aa4-8033e414045c admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.613544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65d18d4-34fd-4231-9aa4-8033e414045c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:38.615542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65d18d4-34fd-4231-9aa4-8033e414045c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.626458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e579f6-7389-4a7e-a750-be69ea112504 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:38.627094 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e579f6-7389-4a7e-a750-be69ea112504 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.627401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e579f6-7389-4a7e-a750-be69ea112504 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.627671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e579f6-7389-4a7e-a750-be69ea112504 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.628382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e579f6-7389-4a7e-a750-be69ea112504 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:38.628687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e579f6-7389-4a7e-a750-be69ea112504 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:38.630346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e579f6-7389-4a7e-a750-be69ea112504 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.646502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1032/2055] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:09:38.648341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e579f6-7389-4a7e-a750-be69ea112504 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.651959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1024/2056] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:38 2022] POST /identity/v3/users => generated 300 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:38.653559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65d18d4-34fd-4231-9aa4-8033e414045c admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.665333 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e579f6-7389-4a7e-a750-be69ea112504 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:38.668981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65d18d4-34fd-4231-9aa4-8033e414045c admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:38.672185 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ab0498-4e5c-4706-9bc2-bc2d30971fcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:38.673126 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ab0498-4e5c-4706-9bc2-bc2d30971fcd admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.673587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ab0498-4e5c-4706-9bc2-bc2d30971fcd admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.673975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ab0498-4e5c-4706-9bc2-bc2d30971fcd admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.674975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ab0498-4e5c-4706-9bc2-bc2d30971fcd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:38.676545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a894d32d-631c-454b-bea2-ee2673206580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:38.677241 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a894d32d-631c-454b-bea2-ee2673206580 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.677430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ab0498-4e5c-4706-9bc2-bc2d30971fcd admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.677569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a894d32d-631c-454b-bea2-ee2673206580 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.677835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a894d32d-631c-454b-bea2-ee2673206580 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.678480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a894d32d-631c-454b-bea2-ee2673206580 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:38.680477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a894d32d-631c-454b-bea2-ee2673206580 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.692370 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1033/2057] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:38 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:38.693670 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ab0498-4e5c-4706-9bc2-bc2d30971fcd admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.695161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1025/2058] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:38 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:38.698726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a894d32d-631c-454b-bea2-ee2673206580 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.711635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ab0498-4e5c-4706-9bc2-bc2d30971fcd admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:38.713549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a894d32d-631c-454b-bea2-ee2673206580 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:38.717481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0064ecdb-02a9-47c5-8614-8881a8558e60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:38.718366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0064ecdb-02a9-47c5-8614-8881a8558e60 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.718607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0064ecdb-02a9-47c5-8614-8881a8558e60 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.718884 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0064ecdb-02a9-47c5-8614-8881a8558e60 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.719761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0064ecdb-02a9-47c5-8614-8881a8558e60 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:38.720659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9118c5b0-57bd-419b-8cfb-8ff1b43c5cde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:38.721606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9118c5b0-57bd-419b-8cfb-8ff1b43c5cde admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.721875 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0064ecdb-02a9-47c5-8614-8881a8558e60 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.722032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9118c5b0-57bd-419b-8cfb-8ff1b43c5cde admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.722543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9118c5b0-57bd-419b-8cfb-8ff1b43c5cde admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.723581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9118c5b0-57bd-419b-8cfb-8ff1b43c5cde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:38.726231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9118c5b0-57bd-419b-8cfb-8ff1b43c5cde admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.742864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1026/2059] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:38 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:38.743749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9118c5b0-57bd-419b-8cfb-8ff1b43c5cde admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.762525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9118c5b0-57bd-419b-8cfb-8ff1b43c5cde admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:38.762802 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1034/2060] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:38 2022] POST /identity/v3/projects => generated 375 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:38.763538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0064ecdb-02a9-47c5-8614-8881a8558e60 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.862799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efb3fa2-4f0c-40d0-8c22-2417eaf49774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:38.863629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efb3fa2-4f0c-40d0-8c22-2417eaf49774 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.863997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efb3fa2-4f0c-40d0-8c22-2417eaf49774 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.864324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efb3fa2-4f0c-40d0-8c22-2417eaf49774 admin admin] PATH_INFO: `/v3/projects/f0b66d9717e24f70800bd6f439f20656/users/b28eacfefc8340b1b2075345227d6eb5/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.891270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0064ecdb-02a9-47c5-8614-8881a8558e60 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:38.896866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9efb3fa2-4f0c-40d0-8c22-2417eaf49774 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0b66d9717e24f70800bd6f439f20656, user_id=b28eacfefc8340b1b2075345227d6eb5, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:38.897142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f762f1a-fe36-4e9e-91a0-8a62c0b21297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:38.897897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f762f1a-fe36-4e9e-91a0-8a62c0b21297 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:38.898401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f762f1a-fe36-4e9e-91a0-8a62c0b21297 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:38.898785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f762f1a-fe36-4e9e-91a0-8a62c0b21297 admin admin] PATH_INFO: `/v3/projects/02877c30378a4d4c978513ad644e6893/users/48790f68a40444aa9e8b5a5f8454e8c6/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:38.900517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9efb3fa2-4f0c-40d0-8c22-2417eaf49774 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.920614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f762f1a-fe36-4e9e-91a0-8a62c0b21297 admin admin] RBAC: Authorizing `identity:create_grant(project_id=02877c30378a4d4c978513ad644e6893, user_id=48790f68a40444aa9e8b5a5f8454e8c6, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:38.924282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f762f1a-fe36-4e9e-91a0-8a62c0b21297 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:38.929930 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1027/2061] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:38 2022] PUT /identity/v3/projects/f0b66d9717e24f70800bd6f439f20656/users/b28eacfefc8340b1b2075345227d6eb5/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:38.930883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efb3fa2-4f0c-40d0-8c22-2417eaf49774 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:38.952036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1035/2062] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:38 2022] PUT /identity/v3/projects/02877c30378a4d4c978513ad644e6893/users/48790f68a40444aa9e8b5a5f8454e8c6/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:38.962192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f762f1a-fe36-4e9e-91a0-8a62c0b21297 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:39.005193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efb3fa2-4f0c-40d0-8c22-2417eaf49774 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:39.038375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:39.067108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f762f1a-fe36-4e9e-91a0-8a62c0b21297 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:39.072440 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:39.073017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f6e7f8-44f0-4726-8011-88dca625678b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:39.073666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f6e7f8-44f0-4726-8011-88dca625678b admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.074330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f6e7f8-44f0-4726-8011-88dca625678b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.074615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f6e7f8-44f0-4726-8011-88dca625678b admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.075580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f6e7f8-44f0-4726-8011-88dca625678b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:39.078702 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f6e7f8-44f0-4726-8011-88dca625678b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:39.088568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e303e0e5-32fb-4390-8d8b-effe1efb8bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:39.089399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303e0e5-32fb-4390-8d8b-effe1efb8bce admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.089787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303e0e5-32fb-4390-8d8b-effe1efb8bce admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.090150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303e0e5-32fb-4390-8d8b-effe1efb8bce admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.090954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e303e0e5-32fb-4390-8d8b-effe1efb8bce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:39.092514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1036/2063] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:38 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:39.093881 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cd2f15-2cf0-4911-916e-eb979b7f4883 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.094165 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cd2f15-2cf0-4911-916e-eb979b7f4883 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.094446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cd2f15-2cf0-4911-916e-eb979b7f4883 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.097156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e303e0e5-32fb-4390-8d8b-effe1efb8bce admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:39.126154 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1028/2064] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:38 2022] POST /identity/v3/users => generated 288 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:39.127828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e303e0e5-32fb-4390-8d8b-effe1efb8bce admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:39.128110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f1cd2f15-2cf0-4911-916e-eb979b7f4883 None None] MFA Rules not processed for user `48790f68a40444aa9e8b5a5f8454e8c6`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:39.130254 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f1cd2f15-2cf0-4911-916e-eb979b7f4883 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:39.145394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e303e0e5-32fb-4390-8d8b-effe1efb8bce admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:39.151022 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb81e7f-44b8-490a-9081-2d773b051f48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:39.151766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb81e7f-44b8-490a-9081-2d773b051f48 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.152050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb81e7f-44b8-490a-9081-2d773b051f48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.152323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb81e7f-44b8-490a-9081-2d773b051f48 admin admin] PATH_INFO: `/v3/projects/f0b66d9717e24f70800bd6f439f20656/users/b28eacfefc8340b1b2075345227d6eb5/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.154454 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb81e7f-44b8-490a-9081-2d773b051f48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0b66d9717e24f70800bd6f439f20656, user_id=b28eacfefc8340b1b2075345227d6eb5, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:39.157332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb81e7f-44b8-490a-9081-2d773b051f48 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:39.170042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1029/2065] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:39 2022] PUT /identity/v3/projects/f0b66d9717e24f70800bd6f439f20656/users/b28eacfefc8340b1b2075345227d6eb5/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:39.184328 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb81e7f-44b8-490a-9081-2d773b051f48 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:39.235963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1037/2066] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:39 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:39.237796 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109ae33f-a55c-4b32-8e8f-b742e2b5cae6 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.238182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109ae33f-a55c-4b32-8e8f-b742e2b5cae6 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.238550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109ae33f-a55c-4b32-8e8f-b742e2b5cae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.280367 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-109ae33f-a55c-4b32-8e8f-b742e2b5cae6 None None] MFA Rules not processed for user `b28eacfefc8340b1b2075345227d6eb5`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:39.280953 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb81e7f-44b8-490a-9081-2d773b051f48 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:39.282412 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-109ae33f-a55c-4b32-8e8f-b742e2b5cae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:39.288525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de959aa5-bdb4-4828-a056-b312a6ef13e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:39.289371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de959aa5-bdb4-4828-a056-b312a6ef13e2 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.289824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de959aa5-bdb4-4828-a056-b312a6ef13e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.290272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de959aa5-bdb4-4828-a056-b312a6ef13e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.291262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de959aa5-bdb4-4828-a056-b312a6ef13e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:39.293289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de959aa5-bdb4-4828-a056-b312a6ef13e2 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:39.307268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1030/2067] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:39 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:39.308659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de959aa5-bdb4-4828-a056-b312a6ef13e2 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:39.324524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de959aa5-bdb4-4828-a056-b312a6ef13e2 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:39.330997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bfbbaa-014e-4d58-a923-c7f9329dfd75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:39.331959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bfbbaa-014e-4d58-a923-c7f9329dfd75 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.332283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bfbbaa-014e-4d58-a923-c7f9329dfd75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.332602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bfbbaa-014e-4d58-a923-c7f9329dfd75 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.333530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bfbbaa-014e-4d58-a923-c7f9329dfd75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:39.336213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bfbbaa-014e-4d58-a923-c7f9329dfd75 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:39.378333 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1031/2068] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:39 2022] POST /identity/v3/projects => generated 383 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:39.379659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bfbbaa-014e-4d58-a923-c7f9329dfd75 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:39.399323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1038/2069] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:39 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:39.404782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109ae33f-a55c-4b32-8e8f-b742e2b5cae6 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:39.419543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109ae33f-a55c-4b32-8e8f-b742e2b5cae6 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:39.485324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bfbbaa-014e-4d58-a923-c7f9329dfd75 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:39.493697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f94bf11-9f30-474d-b0a2-400f61f8cc35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:39.495128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f94bf11-9f30-474d-b0a2-400f61f8cc35 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.495655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f94bf11-9f30-474d-b0a2-400f61f8cc35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.496118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f94bf11-9f30-474d-b0a2-400f61f8cc35 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.497426 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f94bf11-9f30-474d-b0a2-400f61f8cc35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:39.500958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f94bf11-9f30-474d-b0a2-400f61f8cc35 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:39.516470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:39.525120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1032/2070] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:39 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:39.525120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f94bf11-9f30-474d-b0a2-400f61f8cc35 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:39.527775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9f94bf11-9f30-474d-b0a2-400f61f8cc35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:39.541073 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce5ebe0-3ec8-417a-88a1-378223a14ba8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:39.542488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce5ebe0-3ec8-417a-88a1-378223a14ba8 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.543078 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce5ebe0-3ec8-417a-88a1-378223a14ba8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.550193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce5ebe0-3ec8-417a-88a1-378223a14ba8 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.552317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce5ebe0-3ec8-417a-88a1-378223a14ba8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:39.559473 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce5ebe0-3ec8-417a-88a1-378223a14ba8 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:39.597922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1039/2071] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:39 2022] POST /identity/v3/users => generated 298 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:39.599447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce5ebe0-3ec8-417a-88a1-378223a14ba8 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:39.617795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce5ebe0-3ec8-417a-88a1-378223a14ba8 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:39.623671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:39.630174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2089c794-7e2d-432b-ae97-7579b83f7c8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:39.631263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2089c794-7e2d-432b-ae97-7579b83f7c8d admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.631745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2089c794-7e2d-432b-ae97-7579b83f7c8d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.632146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2089c794-7e2d-432b-ae97-7579b83f7c8d admin admin] PATH_INFO: `/v3/projects/14a1550c1fa94b6985a8ae59d35b8ec8/users/e2c4ac1fe934430dae8418ddbc20d25a/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.647738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f94bf11-9f30-474d-b0a2-400f61f8cc35 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:39.654165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5e29db-0454-4954-93c6-12c43597c04a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:39.655346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5e29db-0454-4954-93c6-12c43597c04a service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.656795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5e29db-0454-4954-93c6-12c43597c04a service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.657256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5e29db-0454-4954-93c6-12c43597c04a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.662393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2089c794-7e2d-432b-ae97-7579b83f7c8d admin admin] RBAC: Authorizing `identity:create_grant(project_id=14a1550c1fa94b6985a8ae59d35b8ec8, user_id=e2c4ac1fe934430dae8418ddbc20d25a, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:39.668028 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2089c794-7e2d-432b-ae97-7579b83f7c8d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:39.690561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1040/2072] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:39 2022] PUT /identity/v3/projects/14a1550c1fa94b6985a8ae59d35b8ec8/users/e2c4ac1fe934430dae8418ddbc20d25a/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:39.691998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2089c794-7e2d-432b-ae97-7579b83f7c8d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:39.752207 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5e29db-0454-4954-93c6-12c43597c04a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:39.754301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5e29db-0454-4954-93c6-12c43597c04a service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:39.771895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1033/2073] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:39 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:39.774281 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fcc44-7270-4298-a333-0d858b465472 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.775039 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fcc44-7270-4298-a333-0d858b465472 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.775587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fcc44-7270-4298-a333-0d858b465472 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.827251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2089c794-7e2d-432b-ae97-7579b83f7c8d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:39.828456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-193fcc44-7270-4298-a333-0d858b465472 None None] MFA Rules not processed for user `e2c4ac1fe934430dae8418ddbc20d25a`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:39.830433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-193fcc44-7270-4298-a333-0d858b465472 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:39.834518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05c4c10-0b8f-4ccf-b14a-910d93c865f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:39.835544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05c4c10-0b8f-4ccf-b14a-910d93c865f2 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.836001 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05c4c10-0b8f-4ccf-b14a-910d93c865f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.836450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05c4c10-0b8f-4ccf-b14a-910d93c865f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.837631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05c4c10-0b8f-4ccf-b14a-910d93c865f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:39.840247 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05c4c10-0b8f-4ccf-b14a-910d93c865f2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:39.855180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1041/2074] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:39 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:39.861816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05c4c10-0b8f-4ccf-b14a-910d93c865f2 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:39.877252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05c4c10-0b8f-4ccf-b14a-910d93c865f2 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:39.888572 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf20d69-e906-48f5-a5c4-1107892f10f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:39.889284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf20d69-e906-48f5-a5c4-1107892f10f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:39.889569 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf20d69-e906-48f5-a5c4-1107892f10f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:39.889862 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf20d69-e906-48f5-a5c4-1107892f10f7 admin admin] PATH_INFO: `/v3/projects/97a6b783f1224584b214edbe1c5568f6/users/7df159d06a774c6bac8c6a23799bbe06/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:39.913802 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf20d69-e906-48f5-a5c4-1107892f10f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=97a6b783f1224584b214edbe1c5568f6, user_id=7df159d06a774c6bac8c6a23799bbe06, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:39.916921 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf20d69-e906-48f5-a5c4-1107892f10f7 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:39.960297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1042/2075] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:39 2022] PUT /identity/v3/projects/97a6b783f1224584b214edbe1c5568f6/users/7df159d06a774c6bac8c6a23799bbe06/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:39.965310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf20d69-e906-48f5-a5c4-1107892f10f7 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:40.000063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1034/2076] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:39 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:40.001159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193fcc44-7270-4298-a333-0d858b465472 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:40.015172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193fcc44-7270-4298-a333-0d858b465472 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:40.059473 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf20d69-e906-48f5-a5c4-1107892f10f7 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:40.065056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb94162e-1c03-4eec-bfb2-be23bb5127d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:40.065695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb94162e-1c03-4eec-bfb2-be23bb5127d4 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.065969 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb94162e-1c03-4eec-bfb2-be23bb5127d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.066244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb94162e-1c03-4eec-bfb2-be23bb5127d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.066962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb94162e-1c03-4eec-bfb2-be23bb5127d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:40.068638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb94162e-1c03-4eec-bfb2-be23bb5127d4 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:40.080996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1043/2077] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:39 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:09:40.082057 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb94162e-1c03-4eec-bfb2-be23bb5127d4 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:40.094080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb94162e-1c03-4eec-bfb2-be23bb5127d4 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:40.096120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cac6352-2de0-4eb1-9ba8-004fa645a8b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:40.096849 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cac6352-2de0-4eb1-9ba8-004fa645a8b2 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.097239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cac6352-2de0-4eb1-9ba8-004fa645a8b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.097648 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cac6352-2de0-4eb1-9ba8-004fa645a8b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.098596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cac6352-2de0-4eb1-9ba8-004fa645a8b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:40.099202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:40.099891 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.100175 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.100448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] PATH_INFO: `/v3/users/e448dc9944a243cf931f63514060ada3` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.101101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cac6352-2de0-4eb1-9ba8-004fa645a8b2 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:40.101714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e448dc9944a243cf931f63514060ada3)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:40.103716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:40.138243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1035/2078] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:39 2022] POST /identity/v3/projects => generated 377 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:40.138243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cac6352-2de0-4eb1-9ba8-004fa645a8b2 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:40.141131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:40.147389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:40.180431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e448dc9944a243cf931f63514060ada3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:40.181132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e448dc9944a243cf931f63514060ada3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:40.191001 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e448dc9944a243cf931f63514060ada3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:40.216273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e448dc9944a243cf931f63514060ada3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:40.227892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cac6352-2de0-4eb1-9ba8-004fa645a8b2 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:40.231042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e448dc9944a243cf931f63514060ada3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:40.233024 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b169af-f27f-4b17-bc95-ce0dfc2b2a87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:40.233757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b169af-f27f-4b17-bc95-ce0dfc2b2a87 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.234051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b169af-f27f-4b17-bc95-ce0dfc2b2a87 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.234307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b169af-f27f-4b17-bc95-ce0dfc2b2a87 admin admin] PATH_INFO: `/v3/projects/97a6b783f1224584b214edbe1c5568f6/users/7df159d06a774c6bac8c6a23799bbe06/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.236444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b169af-f27f-4b17-bc95-ce0dfc2b2a87 admin admin] RBAC: Authorizing `identity:create_grant(project_id=97a6b783f1224584b214edbe1c5568f6, user_id=7df159d06a774c6bac8c6a23799bbe06, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:40.239175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b169af-f27f-4b17-bc95-ce0dfc2b2a87 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:40.246132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1044/2079] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:40 2022] DELETE /identity/v3/users/e448dc9944a243cf931f63514060ada3 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:40.246132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:40.248461 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:40.252131 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1036/2080] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:40 2022] PUT /identity/v3/projects/97a6b783f1224584b214edbe1c5568f6/users/7df159d06a774c6bac8c6a23799bbe06/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:40.253574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d208638-ec6d-4b44-94c3-080bdd65e3dd None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.253917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d208638-ec6d-4b44-94c3-080bdd65e3dd None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.254243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d208638-ec6d-4b44-94c3-080bdd65e3dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.282906 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-0d208638-ec6d-4b44-94c3-080bdd65e3dd None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:40.284216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0d208638-ec6d-4b44-94c3-080bdd65e3dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:40.359325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8610a529-819a-4491-9372-92f2fb8e15e6 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:40.364164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfb1e44-3e09-4d41-8723-c699abef37f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:40.364871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfb1e44-3e09-4d41-8723-c699abef37f1 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.365148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfb1e44-3e09-4d41-8723-c699abef37f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.365418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfb1e44-3e09-4d41-8723-c699abef37f1 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.366097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfb1e44-3e09-4d41-8723-c699abef37f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:40.366239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1037/2081] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:40.368155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfb1e44-3e09-4d41-8723-c699abef37f1 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:40.369507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26df944-f400-4ed0-9a63-f0df4f26f8ca None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.370006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26df944-f400-4ed0-9a63-f0df4f26f8ca None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.370837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26df944-f400-4ed0-9a63-f0df4f26f8ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.402307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1045/2082] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:40 2022] POST /identity/v3/users => generated 289 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:40.402307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfb1e44-3e09-4d41-8723-c699abef37f1 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:40.402307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9bfb1e44-3e09-4d41-8723-c699abef37f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:40.422460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c26df944-f400-4ed0-9a63-f0df4f26f8ca None None] MFA Rules not processed for user `7df159d06a774c6bac8c6a23799bbe06`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:40.424679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c26df944-f400-4ed0-9a63-f0df4f26f8ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:40.469630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfb1e44-3e09-4d41-8723-c699abef37f1 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:40.475937 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd98b2da-6dfe-42c9-8f69-773a45dc4ef2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:40.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd98b2da-6dfe-42c9-8f69-773a45dc4ef2 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.477503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd98b2da-6dfe-42c9-8f69-773a45dc4ef2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.478193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd98b2da-6dfe-42c9-8f69-773a45dc4ef2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.496065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd98b2da-6dfe-42c9-8f69-773a45dc4ef2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:40.498991 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd98b2da-6dfe-42c9-8f69-773a45dc4ef2 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:40.518537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1046/2083] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:40.520097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd98b2da-6dfe-42c9-8f69-773a45dc4ef2 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:40.527393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1038/2084] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:40 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:40.539741 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26df944-f400-4ed0-9a63-f0df4f26f8ca None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:40.541187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd98b2da-6dfe-42c9-8f69-773a45dc4ef2 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:40.542545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c26df944-f400-4ed0-9a63-f0df4f26f8ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:40.549974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c60d19-40ea-4680-b278-4ca9c31bf5c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:40.550945 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c60d19-40ea-4680-b278-4ca9c31bf5c5 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.551452 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c60d19-40ea-4680-b278-4ca9c31bf5c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.552014 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c60d19-40ea-4680-b278-4ca9c31bf5c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.553212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c60d19-40ea-4680-b278-4ca9c31bf5c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:40.556433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c60d19-40ea-4680-b278-4ca9c31bf5c5 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:40.586605 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1047/2085] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:40 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:40.592666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c60d19-40ea-4680-b278-4ca9c31bf5c5 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:40.611970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c60d19-40ea-4680-b278-4ca9c31bf5c5 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:40.612231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:40.620649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32532ac-7560-4d3c-92f8-1917986336e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:40.622059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32532ac-7560-4d3c-92f8-1917986336e2 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.622516 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32532ac-7560-4d3c-92f8-1917986336e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.622962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32532ac-7560-4d3c-92f8-1917986336e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.624712 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32532ac-7560-4d3c-92f8-1917986336e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:40.629496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32532ac-7560-4d3c-92f8-1917986336e2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:40.647273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1048/2086] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:40 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:09:40.652928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32532ac-7560-4d3c-92f8-1917986336e2 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:40.673567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32532ac-7560-4d3c-92f8-1917986336e2 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:40.678312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26df944-f400-4ed0-9a63-f0df4f26f8ca None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:40.691842 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e738216-71e3-4126-9c70-33106555940c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:40.693151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e738216-71e3-4126-9c70-33106555940c service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.693722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e738216-71e3-4126-9c70-33106555940c service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.694286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e738216-71e3-4126-9c70-33106555940c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.696227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb1d772-72dc-4a38-8032-36fedf7c069c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:40.697733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb1d772-72dc-4a38-8032-36fedf7c069c admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.698465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb1d772-72dc-4a38-8032-36fedf7c069c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.699328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb1d772-72dc-4a38-8032-36fedf7c069c admin admin] PATH_INFO: `/v3/projects/784de17a2e9040fe8aba15604cdcc935/users/6d4bf25fd67749209f8905737c11810a/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.706013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:40.716437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e738216-71e3-4126-9c70-33106555940c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:40.720389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e738216-71e3-4126-9c70-33106555940c service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:40.739704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb1d772-72dc-4a38-8032-36fedf7c069c admin admin] RBAC: Authorizing `identity:create_grant(project_id=784de17a2e9040fe8aba15604cdcc935, user_id=6d4bf25fd67749209f8905737c11810a, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:40.744049 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb1d772-72dc-4a38-8032-36fedf7c069c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:40.745129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1039/2087] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:40 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:09:40.777698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1049/2088] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:40 2022] PUT /identity/v3/projects/784de17a2e9040fe8aba15604cdcc935/users/6d4bf25fd67749209f8905737c11810a/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:40.795777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f00c5c-a8a9-44be-99d6-d6fbe0fad819 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:40.797234 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f00c5c-a8a9-44be-99d6-d6fbe0fad819 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:40.797234 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f00c5c-a8a9-44be-99d6-d6fbe0fad819 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:40.855709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-51f00c5c-a8a9-44be-99d6-d6fbe0fad819 None None] MFA Rules not processed for user `6d4bf25fd67749209f8905737c11810a`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:40.858242 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-51f00c5c-a8a9-44be-99d6-d6fbe0fad819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:40.918642 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb1d772-72dc-4a38-8032-36fedf7c069c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:41.000212 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1040/2089] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:40 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:09:41.005473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f00c5c-a8a9-44be-99d6-d6fbe0fad819 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:41.052059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb1d772-72dc-4a38-8032-36fedf7c069c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:41.056731 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e7d1c6-2968-4fb0-9d42-877236b39a99 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:41.057371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e7d1c6-2968-4fb0-9d42-877236b39a99 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:41.057650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e7d1c6-2968-4fb0-9d42-877236b39a99 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:41.058117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e7d1c6-2968-4fb0-9d42-877236b39a99 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:41.061059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b7e7d1c6-2968-4fb0-9d42-877236b39a99 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:41.139384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f00c5c-a8a9-44be-99d6-d6fbe0fad819 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:41.145349 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a39e1b-d31e-4b2f-bc1f-bd91605399ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:41.146165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a39e1b-d31e-4b2f-bc1f-bd91605399ed admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:41.146442 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a39e1b-d31e-4b2f-bc1f-bd91605399ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:41.146840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a39e1b-d31e-4b2f-bc1f-bd91605399ed admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:41.147655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a39e1b-d31e-4b2f-bc1f-bd91605399ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:41.150005 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a39e1b-d31e-4b2f-bc1f-bd91605399ed admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:41.195612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e7d1c6-2968-4fb0-9d42-877236b39a99 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:41.197040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1041/2090] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:41 2022] POST /identity/v3/projects => generated 377 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:41.197355 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e7d1c6-2968-4fb0-9d42-877236b39a99 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:41.201120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a39e1b-d31e-4b2f-bc1f-bd91605399ed admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:41.211862 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1050/2091] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:40 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:41.231102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e7d1c6-2968-4fb0-9d42-877236b39a99 service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:41.233637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b7e7d1c6-2968-4fb0-9d42-877236b39a99 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:41.269324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e7d1c6-2968-4fb0-9d42-877236b39a99 service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:41.313194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:41.347311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a39e1b-d31e-4b2f-bc1f-bd91605399ed admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:41.356873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1068b29b-1a89-4d77-befc-a36afba52ccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:41.357904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1068b29b-1a89-4d77-befc-a36afba52ccd admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:41.358364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1068b29b-1a89-4d77-befc-a36afba52ccd admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:41.358744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1068b29b-1a89-4d77-befc-a36afba52ccd admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:41.360838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1068b29b-1a89-4d77-befc-a36afba52ccd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:41.363406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1068b29b-1a89-4d77-befc-a36afba52ccd admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:41.371625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:41.372709 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:41.373970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:41.374987 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] PATH_INFO: `/v3/projects/5f426702722f491594bb883e4bcd6776` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:41.377481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f426702722f491594bb883e4bcd6776)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:41.383399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:41.404828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1042/2092] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:41 2022] POST /identity/v3/users => generated 295 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:41.406704 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1068b29b-1a89-4d77-befc-a36afba52ccd admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:41.424449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1068b29b-1a89-4d77-befc-a36afba52ccd admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:41.432784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f4d4de-9fc2-48d4-a641-fed59330137e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:41.433697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f4d4de-9fc2-48d4-a641-fed59330137e admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:41.434427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f4d4de-9fc2-48d4-a641-fed59330137e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:41.434873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f4d4de-9fc2-48d4-a641-fed59330137e admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:41.436175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f4d4de-9fc2-48d4-a641-fed59330137e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:41.439663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f4d4de-9fc2-48d4-a641-fed59330137e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:41.457044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:41.461374 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1043/2093] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:41 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:09:41.467572 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f4d4de-9fc2-48d4-a641-fed59330137e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:41.470758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f426702722f491594bb883e4bcd6776', 'request_id': 'req-3a1c0361-cb59-4f59-856f-7c4eb77dded0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:41.485786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f4d4de-9fc2-48d4-a641-fed59330137e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:41.487031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f426702722f491594bb883e4bcd6776', 'request_id': 'req-3a1c0361-cb59-4f59-856f-7c4eb77dded0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:41.498966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f426702722f491594bb883e4bcd6776', 'request_id': 'req-3a1c0361-cb59-4f59-856f-7c4eb77dded0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:41.524783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] The token cache is being invalidate because project 5f426702722f491594bb883e4bcd6776 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:41.525689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:41.526591 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:41.529299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1051/2094] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:41 2022] DELETE /identity/v3/projects/5f426702722f491594bb883e4bcd6776 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:41.535525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:41.537818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:41.600947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573bd56a-8ac2-4daa-bf80-dfee1b223468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:41.601683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573bd56a-8ac2-4daa-bf80-dfee1b223468 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:41.601953 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573bd56a-8ac2-4daa-bf80-dfee1b223468 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:41.602233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573bd56a-8ac2-4daa-bf80-dfee1b223468 admin admin] PATH_INFO: `/v3/projects/55e30b5482384ba7902f98657f155a68/users/65e17774b06a480f90d4c627ca8f587f/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:41.632254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573bd56a-8ac2-4daa-bf80-dfee1b223468 admin admin] RBAC: Authorizing `identity:create_grant(project_id=55e30b5482384ba7902f98657f155a68, user_id=65e17774b06a480f90d4c627ca8f587f, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:41.637181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573bd56a-8ac2-4daa-bf80-dfee1b223468 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:41.661733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1c0361-cb59-4f59-856f-7c4eb77dded0 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:41.663406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1044/2095] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:41 2022] PUT /identity/v3/projects/55e30b5482384ba7902f98657f155a68/users/65e17774b06a480f90d4c627ca8f587f/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:41.668563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573bd56a-8ac2-4daa-bf80-dfee1b223468 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:41.670501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-573bd56a-8ac2-4daa-bf80-dfee1b223468 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:41.794562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87862f98-0668-4205-b839-5b644efd940d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:41.795649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87862f98-0668-4205-b839-5b644efd940d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:41.796015 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87862f98-0668-4205-b839-5b644efd940d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:41.797025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87862f98-0668-4205-b839-5b644efd940d admin admin] PATH_INFO: `/v3/users/b28eacfefc8340b1b2075345227d6eb5` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:41.799215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87862f98-0668-4205-b839-5b644efd940d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b28eacfefc8340b1b2075345227d6eb5)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:41.802878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87862f98-0668-4205-b839-5b644efd940d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:41.848001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573bd56a-8ac2-4daa-bf80-dfee1b223468 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:41.857043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e27c8d-b888-4609-9c7d-cef280bff931 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:41.857764 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e27c8d-b888-4609-9c7d-cef280bff931 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:41.859122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e27c8d-b888-4609-9c7d-cef280bff931 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:41.859494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e27c8d-b888-4609-9c7d-cef280bff931 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:41.860441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e27c8d-b888-4609-9c7d-cef280bff931 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:41.864026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e27c8d-b888-4609-9c7d-cef280bff931 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:41.884755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1045/2096] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:41 2022] GET /identity/v3/roles => generated 1420 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:41.888883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e27c8d-b888-4609-9c7d-cef280bff931 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:41.890869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87862f98-0668-4205-b839-5b644efd940d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b28eacfefc8340b1b2075345227d6eb5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:41.891799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87862f98-0668-4205-b839-5b644efd940d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b28eacfefc8340b1b2075345227d6eb5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:41.902457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e27c8d-b888-4609-9c7d-cef280bff931 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:41.902719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87862f98-0668-4205-b839-5b644efd940d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b28eacfefc8340b1b2075345227d6eb5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:41.911129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7bf51c-7126-413b-bfaa-398eec40aafb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:41.911913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7bf51c-7126-413b-bfaa-398eec40aafb admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:41.912197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7bf51c-7126-413b-bfaa-398eec40aafb admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:41.912539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7bf51c-7126-413b-bfaa-398eec40aafb admin admin] PATH_INFO: `/v3/projects/55e30b5482384ba7902f98657f155a68/users/65e17774b06a480f90d4c627ca8f587f/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:41.915217 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7bf51c-7126-413b-bfaa-398eec40aafb admin admin] RBAC: Authorizing `identity:create_grant(project_id=55e30b5482384ba7902f98657f155a68, user_id=65e17774b06a480f90d4c627ca8f587f, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:41.923957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87862f98-0668-4205-b839-5b644efd940d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b28eacfefc8340b1b2075345227d6eb5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:41.927631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7bf51c-7126-413b-bfaa-398eec40aafb admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:41.931442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:41.934064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87862f98-0668-4205-b839-5b644efd940d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b28eacfefc8340b1b2075345227d6eb5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:41.943477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1052/2097] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:41 2022] DELETE /identity/v3/users/b28eacfefc8340b1b2075345227d6eb5 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:41.957466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1046/2098] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:41 2022] PUT /identity/v3/projects/55e30b5482384ba7902f98657f155a68/users/65e17774b06a480f90d4c627ca8f587f/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:41.971980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8939fd75-0b22-44df-8906-bc5c64badd76 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:41.972704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8939fd75-0b22-44df-8906-bc5c64badd76 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:41.973357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8939fd75-0b22-44df-8906-bc5c64badd76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:42.008929 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-8939fd75-0b22-44df-8906-bc5c64badd76 None None] MFA Rules not processed for user `65e17774b06a480f90d4c627ca8f587f`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:42.010172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8939fd75-0b22-44df-8906-bc5c64badd76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:42.125086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1053/2099] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:41 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:42.143563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7bf51c-7126-413b-bfaa-398eec40aafb admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:42.148991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-cc7bf51c-7126-413b-bfaa-398eec40aafb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:42.214392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:42.250475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7bf51c-7126-413b-bfaa-398eec40aafb admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:42.255471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736df83e-f3b9-4749-ae93-529326b68153 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:42.256287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736df83e-f3b9-4749-ae93-529326b68153 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:42.256671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736df83e-f3b9-4749-ae93-529326b68153 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:42.257139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736df83e-f3b9-4749-ae93-529326b68153 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:42.259568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-736df83e-f3b9-4749-ae93-529326b68153 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:42.366322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736df83e-f3b9-4749-ae93-529326b68153 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:42.371008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736df83e-f3b9-4749-ae93-529326b68153 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:42.381694 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8939fd75-0b22-44df-8906-bc5c64badd76 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:42.383998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8939fd75-0b22-44df-8906-bc5c64badd76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:42.394554 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1047/2100] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:42 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 254 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:42.531585 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8939fd75-0b22-44df-8906-bc5c64badd76 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:42.539242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f845b1-b702-4d3f-891e-b9164e18684f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:42.540056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f845b1-b702-4d3f-891e-b9164e18684f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:42.540421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f845b1-b702-4d3f-891e-b9164e18684f admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:42.540686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f845b1-b702-4d3f-891e-b9164e18684f admin admin] PATH_INFO: `/v3/projects/f0b66d9717e24f70800bd6f439f20656` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:42.542150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f845b1-b702-4d3f-891e-b9164e18684f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0b66d9717e24f70800bd6f439f20656)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:42.545577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f845b1-b702-4d3f-891e-b9164e18684f admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:42.632109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e4f845b1-b702-4d3f-891e-b9164e18684f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0b66d9717e24f70800bd6f439f20656', 'request_id': 'req-e4f845b1-b702-4d3f-891e-b9164e18684f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:42.647223 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e4f845b1-b702-4d3f-891e-b9164e18684f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0b66d9717e24f70800bd6f439f20656', 'request_id': 'req-e4f845b1-b702-4d3f-891e-b9164e18684f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:42.657890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e4f845b1-b702-4d3f-891e-b9164e18684f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0b66d9717e24f70800bd6f439f20656', 'request_id': 'req-e4f845b1-b702-4d3f-891e-b9164e18684f'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:42.677839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e4f845b1-b702-4d3f-891e-b9164e18684f admin admin] The token cache is being invalidate because project f0b66d9717e24f70800bd6f439f20656 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:42.678111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e4f845b1-b702-4d3f-891e-b9164e18684f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:42.679019 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e4f845b1-b702-4d3f-891e-b9164e18684f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:42.681002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1054/2101] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:42 2022] DELETE /identity/v3/projects/f0b66d9717e24f70800bd6f439f20656 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:42.686250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2597083e-0c46-4050-97f6-36101b49fb76 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:42.686449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2597083e-0c46-4050-97f6-36101b49fb76 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:42.686656 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2597083e-0c46-4050-97f6-36101b49fb76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:42.714932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-2597083e-0c46-4050-97f6-36101b49fb76 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:42.716204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2597083e-0c46-4050-97f6-36101b49fb76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:42.832703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1048/2102] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:09:42.843285 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309f40ea-f1d0-42f1-b397-c560b0e596ef None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:42.844071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309f40ea-f1d0-42f1-b397-c560b0e596ef None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:42.844669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309f40ea-f1d0-42f1-b397-c560b0e596ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:42.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-309f40ea-f1d0-42f1-b397-c560b0e596ef None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:42.890993 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-309f40ea-f1d0-42f1-b397-c560b0e596ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:42.910072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1055/2103] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:42.915295 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2597083e-0c46-4050-97f6-36101b49fb76 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:42.945327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2597083e-0c46-4050-97f6-36101b49fb76 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:42.958504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7576b572-d244-4c34-805d-a5bb6e3a5e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:42.959642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7576b572-d244-4c34-805d-a5bb6e3a5e07 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:42.960339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7576b572-d244-4c34-805d-a5bb6e3a5e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:42.960830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7576b572-d244-4c34-805d-a5bb6e3a5e07 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:42.961985 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7576b572-d244-4c34-805d-a5bb6e3a5e07 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:42.962520 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7576b572-d244-4c34-805d-a5bb6e3a5e07 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:42.965533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7576b572-d244-4c34-805d-a5bb6e3a5e07 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:42.988655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1049/2104] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:09:43.000372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309f40ea-f1d0-42f1-b397-c560b0e596ef None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:43.010681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:43.026486 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309f40ea-f1d0-42f1-b397-c560b0e596ef None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:43.033438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c31daa-dcaf-4531-96db-54bb858956f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:43.034415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c31daa-dcaf-4531-96db-54bb858956f8 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:43.034809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c31daa-dcaf-4531-96db-54bb858956f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:43.035250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c31daa-dcaf-4531-96db-54bb858956f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:43.036123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c31daa-dcaf-4531-96db-54bb858956f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:43.038895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c31daa-dcaf-4531-96db-54bb858956f8 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:43.087554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1056/2105] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:42 2022] POST /identity/v3/projects => generated 381 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:43.091859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7576b572-d244-4c34-805d-a5bb6e3a5e07 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:43.103790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7576b572-d244-4c34-805d-a5bb6e3a5e07 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:43.203830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72afbc89-becd-4c89-8c34-08e353248e47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:43.204829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72afbc89-becd-4c89-8c34-08e353248e47 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:43.205328 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72afbc89-becd-4c89-8c34-08e353248e47 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:43.205862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72afbc89-becd-4c89-8c34-08e353248e47 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:43.206998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72afbc89-becd-4c89-8c34-08e353248e47 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:43.212059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72afbc89-becd-4c89-8c34-08e353248e47 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:43.243979 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:43.255394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1050/2106] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:43 2022] POST /identity/v3/users => generated 291 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:43.261475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c31daa-dcaf-4531-96db-54bb858956f8 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:43.275303 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c31daa-dcaf-4531-96db-54bb858956f8 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:43.281114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb1cae8-bfdc-4d4f-a5d0-13481d90a1b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:43.282046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1cae8-bfdc-4d4f-a5d0-13481d90a1b0 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:43.282333 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1cae8-bfdc-4d4f-a5d0-13481d90a1b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:43.282652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1cae8-bfdc-4d4f-a5d0-13481d90a1b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:43.283594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb1cae8-bfdc-4d4f-a5d0-13481d90a1b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:43.285733 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb1cae8-bfdc-4d4f-a5d0-13481d90a1b0 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:43.307130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1057/2107] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:43 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:43.312565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72afbc89-becd-4c89-8c34-08e353248e47 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:43.323743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72afbc89-becd-4c89-8c34-08e353248e47 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:43.329013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82834e62-0a40-4d6d-9b49-7e8c4247edd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:43.329641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82834e62-0a40-4d6d-9b49-7e8c4247edd4 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:43.329923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82834e62-0a40-4d6d-9b49-7e8c4247edd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:43.330194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82834e62-0a40-4d6d-9b49-7e8c4247edd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:43.330903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82834e62-0a40-4d6d-9b49-7e8c4247edd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:43.332590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82834e62-0a40-4d6d-9b49-7e8c4247edd4 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:43.348880 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1051/2108] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:43 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:43.353856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb1cae8-bfdc-4d4f-a5d0-13481d90a1b0 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:43.365580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb1cae8-bfdc-4d4f-a5d0-13481d90a1b0 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:43.370766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b5d79-fcd0-4e92-ac63-981a380aec0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:43.371499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b5d79-fcd0-4e92-ac63-981a380aec0d admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:43.371798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b5d79-fcd0-4e92-ac63-981a380aec0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:43.372081 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b5d79-fcd0-4e92-ac63-981a380aec0d admin admin] PATH_INFO: `/v3/projects/81b988f0484e49e3882bf44913a72861/users/28a7e420909c4984a824830c870396f0/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:43.389617 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567b5d79-fcd0-4e92-ac63-981a380aec0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=81b988f0484e49e3882bf44913a72861, user_id=28a7e420909c4984a824830c870396f0, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:43.392903 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567b5d79-fcd0-4e92-ac63-981a380aec0d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:43.408782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1058/2109] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:43 2022] PUT /identity/v3/projects/81b988f0484e49e3882bf44913a72861/users/28a7e420909c4984a824830c870396f0/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:43.420503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc96c926-c071-454f-8dbf-9964a4d36b88 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:43.421025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc96c926-c071-454f-8dbf-9964a4d36b88 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:43.421555 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc96c926-c071-454f-8dbf-9964a4d36b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:43.459791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-cc96c926-c071-454f-8dbf-9964a4d36b88 None None] MFA Rules not processed for user `28a7e420909c4984a824830c870396f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:43.461397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-cc96c926-c071-454f-8dbf-9964a4d36b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:43.542412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1052/2110] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:43 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:43.546583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b5d79-fcd0-4e92-ac63-981a380aec0d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:43.651746 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b5d79-fcd0-4e92-ac63-981a380aec0d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:43.656709 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417e4d80-1020-4258-9d81-d892aad32b12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:43.657341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417e4d80-1020-4258-9d81-d892aad32b12 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:43.657644 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417e4d80-1020-4258-9d81-d892aad32b12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:43.657899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417e4d80-1020-4258-9d81-d892aad32b12 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:43.658584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417e4d80-1020-4258-9d81-d892aad32b12 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:43.660620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417e4d80-1020-4258-9d81-d892aad32b12 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:43.695720 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1059/2111] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:43 2022] POST /identity/v3/projects => generated 381 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:43.700221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc96c926-c071-454f-8dbf-9964a4d36b88 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:43.713304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc96c926-c071-454f-8dbf-9964a4d36b88 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:43.780193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33750a5-ffa4-47b3-bdd3-f90057782e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:43.780868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33750a5-ffa4-47b3-bdd3-f90057782e49 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:43.781146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33750a5-ffa4-47b3-bdd3-f90057782e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:43.781419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33750a5-ffa4-47b3-bdd3-f90057782e49 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:43.782107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33750a5-ffa4-47b3-bdd3-f90057782e49 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:43.784153 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33750a5-ffa4-47b3-bdd3-f90057782e49 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:43.808502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1053/2112] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:43 2022] POST /identity/v3/users => generated 297 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:09:43.814184 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417e4d80-1020-4258-9d81-d892aad32b12 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:43.825532 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417e4d80-1020-4258-9d81-d892aad32b12 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:43.830462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fe33f8-74ac-40de-9911-9414d78a1247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:43.831093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fe33f8-74ac-40de-9911-9414d78a1247 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:43.831398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fe33f8-74ac-40de-9911-9414d78a1247 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:43.831674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fe33f8-74ac-40de-9911-9414d78a1247 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:43.832385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fe33f8-74ac-40de-9911-9414d78a1247 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:43.834016 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fe33f8-74ac-40de-9911-9414d78a1247 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:43.846411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1060/2113] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:43 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:43.850902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33750a5-ffa4-47b3-bdd3-f90057782e49 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:43.862828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33750a5-ffa4-47b3-bdd3-f90057782e49 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:43.868025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42188e5-6351-419f-b6f7-c7e46964318e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:43.868737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42188e5-6351-419f-b6f7-c7e46964318e admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:43.869015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42188e5-6351-419f-b6f7-c7e46964318e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:43.869300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42188e5-6351-419f-b6f7-c7e46964318e admin admin] PATH_INFO: `/v3/projects/daaac685fc3d4501b5711d45088a4774/users/61acb7201475441aa8b1b88d693cce1e/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:43.888289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42188e5-6351-419f-b6f7-c7e46964318e admin admin] RBAC: Authorizing `identity:create_grant(project_id=daaac685fc3d4501b5711d45088a4774, user_id=61acb7201475441aa8b1b88d693cce1e, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:43.891141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42188e5-6351-419f-b6f7-c7e46964318e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:43.907015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1054/2114] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:43 2022] PUT /identity/v3/projects/daaac685fc3d4501b5711d45088a4774/users/61acb7201475441aa8b1b88d693cce1e/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:43.912119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fe33f8-74ac-40de-9911-9414d78a1247 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:43.930350 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fe33f8-74ac-40de-9911-9414d78a1247 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:44.009298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f6fa7b-6cf0-4678-8a3b-3db46a18e7af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:44.009947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f6fa7b-6cf0-4678-8a3b-3db46a18e7af admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:44.010225 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f6fa7b-6cf0-4678-8a3b-3db46a18e7af admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:44.010500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f6fa7b-6cf0-4678-8a3b-3db46a18e7af admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:44.011230 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f6fa7b-6cf0-4678-8a3b-3db46a18e7af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:44.013332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f6fa7b-6cf0-4678-8a3b-3db46a18e7af admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:44.044558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1061/2115] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:43 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:44.046570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42188e5-6351-419f-b6f7-c7e46964318e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:44.066864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42188e5-6351-419f-b6f7-c7e46964318e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:44.073578 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bd7cc6-5322-4502-b634-fda41f453e6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:44.074278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bd7cc6-5322-4502-b634-fda41f453e6b admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:44.074557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bd7cc6-5322-4502-b634-fda41f453e6b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:44.074905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bd7cc6-5322-4502-b634-fda41f453e6b admin admin] PATH_INFO: `/v3/projects/daaac685fc3d4501b5711d45088a4774/users/61acb7201475441aa8b1b88d693cce1e/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:44.077512 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bd7cc6-5322-4502-b634-fda41f453e6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=daaac685fc3d4501b5711d45088a4774, user_id=61acb7201475441aa8b1b88d693cce1e, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:44.082201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bd7cc6-5322-4502-b634-fda41f453e6b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:44.105738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1055/2116] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:44 2022] PUT /identity/v3/projects/daaac685fc3d4501b5711d45088a4774/users/61acb7201475441aa8b1b88d693cce1e/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:44.126609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e311e41a-5765-4139-a8b2-b9b7e14e668e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:44.126609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e311e41a-5765-4139-a8b2-b9b7e14e668e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:44.126609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e311e41a-5765-4139-a8b2-b9b7e14e668e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:44.169730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-e311e41a-5765-4139-a8b2-b9b7e14e668e None None] MFA Rules not processed for user `61acb7201475441aa8b1b88d693cce1e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:44.172182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e311e41a-5765-4139-a8b2-b9b7e14e668e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:44.180880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:44.290888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1062/2117] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:44 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:09:44.349373 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bd7cc6-5322-4502-b634-fda41f453e6b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:44.351918 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-30bd7cc6-5322-4502-b634-fda41f453e6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:44.412312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:44.447322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bd7cc6-5322-4502-b634-fda41f453e6b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:44.456079 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621059af-74c2-4b81-9d08-17d854d13898 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:44.457640 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621059af-74c2-4b81-9d08-17d854d13898 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:44.458142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621059af-74c2-4b81-9d08-17d854d13898 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:44.458642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621059af-74c2-4b81-9d08-17d854d13898 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:44.484722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621059af-74c2-4b81-9d08-17d854d13898 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:44.488738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621059af-74c2-4b81-9d08-17d854d13898 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:44.511403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1056/2118] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:44 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:45.496572 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e311e41a-5765-4139-a8b2-b9b7e14e668e None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:45.503133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621059af-74c2-4b81-9d08-17d854d13898 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:45.524116 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621059af-74c2-4b81-9d08-17d854d13898 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:45.541599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4304ee04-5a31-49bf-9b39-77fcf4c1f547 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:45.546769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4304ee04-5a31-49bf-9b39-77fcf4c1f547 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:45.547371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4304ee04-5a31-49bf-9b39-77fcf4c1f547 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:45.547861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4304ee04-5a31-49bf-9b39-77fcf4c1f547 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:45.561616 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-4304ee04-5a31-49bf-9b39-77fcf4c1f547 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:45.641191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e311e41a-5765-4139-a8b2-b9b7e14e668e None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:45.648809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:45.649845 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:45.650310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:45.650822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] PATH_INFO: `/v3/users/28a7e420909c4984a824830c870396f0` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:45.653455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] RBAC: Authorizing `identity:delete_user(user_id=28a7e420909c4984a824830c870396f0)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:45.657462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:45.693814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4304ee04-5a31-49bf-9b39-77fcf4c1f547 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:45.696574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4304ee04-5a31-49bf-9b39-77fcf4c1f547 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:45.718054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1057/2119] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:45 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:45.757161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28a7e420909c4984a824830c870396f0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:45.758098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28a7e420909c4984a824830c870396f0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:45.770488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28a7e420909c4984a824830c870396f0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:45.798866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28a7e420909c4984a824830c870396f0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:45.816964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28a7e420909c4984a824830c870396f0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:45.829809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1063/2120] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:45 2022] DELETE /identity/v3/users/28a7e420909c4984a824830c870396f0 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:45.836995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403d5eb6-0f29-4e43-ae36-9b41870ab4aa None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:45.837497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403d5eb6-0f29-4e43-ae36-9b41870ab4aa None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:45.838015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403d5eb6-0f29-4e43-ae36-9b41870ab4aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:45.876826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-403d5eb6-0f29-4e43-ae36-9b41870ab4aa None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:45.876826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-403d5eb6-0f29-4e43-ae36-9b41870ab4aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:45.979960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1058/2121] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:45.997968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:45.999608 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:46.033920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:46.084474 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cfcb5b-1456-4283-b4f9-5449b1ab4d3f admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:46.090683 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08ed023-92aa-40d2-8396-c53eec76231b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:46.091718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08ed023-92aa-40d2-8396-c53eec76231b service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:46.092187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08ed023-92aa-40d2-8396-c53eec76231b service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:46.092643 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08ed023-92aa-40d2-8396-c53eec76231b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:46.109376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08ed023-92aa-40d2-8396-c53eec76231b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:46.111712 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08ed023-92aa-40d2-8396-c53eec76231b service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:46.126882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1064/2122] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:46.467598 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403d5eb6-0f29-4e43-ae36-9b41870ab4aa None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:46.470124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-403d5eb6-0f29-4e43-ae36-9b41870ab4aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:46.494543 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403d5eb6-0f29-4e43-ae36-9b41870ab4aa None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:46.503748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:46.504872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:46.505260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:46.505641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] PATH_INFO: `/v3/projects/81b988f0484e49e3882bf44913a72861` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:46.507179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=81b988f0484e49e3882bf44913a72861)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:46.509887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:46.572970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81b988f0484e49e3882bf44913a72861', 'request_id': 'req-e424f9bc-d747-4a19-a2d5-305593b1d3c0'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:46.586546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81b988f0484e49e3882bf44913a72861', 'request_id': 'req-e424f9bc-d747-4a19-a2d5-305593b1d3c0'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:46.595616 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81b988f0484e49e3882bf44913a72861', 'request_id': 'req-e424f9bc-d747-4a19-a2d5-305593b1d3c0'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:46.600723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:46.614323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] The token cache is being invalidate because project 81b988f0484e49e3882bf44913a72861 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:46.614744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:46.615679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:46.617845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1059/2123] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:46 2022] DELETE /identity/v3/projects/81b988f0484e49e3882bf44913a72861 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:46.622100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08ed023-92aa-40d2-8396-c53eec76231b service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:46.633761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08ed023-92aa-40d2-8396-c53eec76231b service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:46.732774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f07d79-7786-45bb-a4db-4dfcc67a7725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:46.733582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f07d79-7786-45bb-a4db-4dfcc67a7725 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:46.733953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f07d79-7786-45bb-a4db-4dfcc67a7725 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:46.734406 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f07d79-7786-45bb-a4db-4dfcc67a7725 admin admin] PATH_INFO: `/v3/users/61acb7201475441aa8b1b88d693cce1e` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:46.736104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f07d79-7786-45bb-a4db-4dfcc67a7725 admin admin] RBAC: Authorizing `identity:delete_user(user_id=61acb7201475441aa8b1b88d693cce1e)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:46.738152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f07d79-7786-45bb-a4db-4dfcc67a7725 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:46.797195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-63f07d79-7786-45bb-a4db-4dfcc67a7725 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '61acb7201475441aa8b1b88d693cce1e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:46.797939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-63f07d79-7786-45bb-a4db-4dfcc67a7725 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '61acb7201475441aa8b1b88d693cce1e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:46.806675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-63f07d79-7786-45bb-a4db-4dfcc67a7725 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '61acb7201475441aa8b1b88d693cce1e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:46.822047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-63f07d79-7786-45bb-a4db-4dfcc67a7725 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '61acb7201475441aa8b1b88d693cce1e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:46.830448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-63f07d79-7786-45bb-a4db-4dfcc67a7725 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '61acb7201475441aa8b1b88d693cce1e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:46.838425 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1065/2124] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:46 2022] DELETE /identity/v3/users/61acb7201475441aa8b1b88d693cce1e => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:47.530679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:47.532959 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:47.552287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e424f9bc-d747-4a19-a2d5-305593b1d3c0 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:47.638803 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:47.639544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:47.640178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:47.640457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6 admin admin] PATH_INFO: `/v3/projects/daaac685fc3d4501b5711d45088a4774` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:47.641785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=daaac685fc3d4501b5711d45088a4774)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:47.643862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:47.703793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'daaac685fc3d4501b5711d45088a4774', 'request_id': 'req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:47.710869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'daaac685fc3d4501b5711d45088a4774', 'request_id': 'req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:47.717751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'daaac685fc3d4501b5711d45088a4774', 'request_id': 'req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:47.733909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6 admin admin] The token cache is being invalidate because project daaac685fc3d4501b5711d45088a4774 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:47.734198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:47.734807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bbdf9e5b-f49f-4d0b-98e6-cd8304f426d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:47.736396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1060/2125] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:47 2022] DELETE /identity/v3/projects/daaac685fc3d4501b5711d45088a4774 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:47.742909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfccacc-2442-4bb3-b32e-8c7a3f09a7f6 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:47.743549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfccacc-2442-4bb3-b32e-8c7a3f09a7f6 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:47.743899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfccacc-2442-4bb3-b32e-8c7a3f09a7f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:47.777968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ecfccacc-2442-4bb3-b32e-8c7a3f09a7f6 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:47.780431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ecfccacc-2442-4bb3-b32e-8c7a3f09a7f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:47.834965 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:47.861232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1066/2126] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:47.869933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9080db7-09f2-46d9-a0fa-d6c068454d8f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:47.870414 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9080db7-09f2-46d9-a0fa-d6c068454d8f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:47.870751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9080db7-09f2-46d9-a0fa-d6c068454d8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:47.893662 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b9080db7-09f2-46d9-a0fa-d6c068454d8f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:47.894958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b9080db7-09f2-46d9-a0fa-d6c068454d8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:47.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1061/2127] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:09:47.908790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfccacc-2442-4bb3-b32e-8c7a3f09a7f6 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:47.920018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfccacc-2442-4bb3-b32e-8c7a3f09a7f6 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:47.925447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce62784-663c-498a-91fd-8a8ac170bcc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:47.926132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce62784-663c-498a-91fd-8a8ac170bcc0 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:47.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce62784-663c-498a-91fd-8a8ac170bcc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:47.926784 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce62784-663c-498a-91fd-8a8ac170bcc0 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:47.927580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce62784-663c-498a-91fd-8a8ac170bcc0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:47.927924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce62784-663c-498a-91fd-8a8ac170bcc0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:47.929594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce62784-663c-498a-91fd-8a8ac170bcc0 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:47.944578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1067/2128] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:09:47.949770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9080db7-09f2-46d9-a0fa-d6c068454d8f None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:47.967038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9080db7-09f2-46d9-a0fa-d6c068454d8f None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:47.974633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6677e32a-947a-4e90-8b1f-61c7d668dcdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:47.975673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6677e32a-947a-4e90-8b1f-61c7d668dcdc admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:47.976129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6677e32a-947a-4e90-8b1f-61c7d668dcdc admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:47.976573 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6677e32a-947a-4e90-8b1f-61c7d668dcdc admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:47.977694 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6677e32a-947a-4e90-8b1f-61c7d668dcdc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:47.981302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6677e32a-947a-4e90-8b1f-61c7d668dcdc admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:48.031746 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1062/2129] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:47 2022] POST /identity/v3/projects => generated 385 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:09:48.033695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce62784-663c-498a-91fd-8a8ac170bcc0 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:48.046756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce62784-663c-498a-91fd-8a8ac170bcc0 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:48.136341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136676c2-e627-46a4-a239-acb616381007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:48.136979 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136676c2-e627-46a4-a239-acb616381007 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:48.137302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136676c2-e627-46a4-a239-acb616381007 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:48.137617 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136676c2-e627-46a4-a239-acb616381007 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:48.138442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136676c2-e627-46a4-a239-acb616381007 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:48.141304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136676c2-e627-46a4-a239-acb616381007 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:48.166547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1068/2130] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:48 2022] POST /identity/v3/users => generated 293 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:09:48.171073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6677e32a-947a-4e90-8b1f-61c7d668dcdc admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:48.183984 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6677e32a-947a-4e90-8b1f-61c7d668dcdc admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:48.190274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862ddbac-eaef-414b-9d4b-fd327626422a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:48.191073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862ddbac-eaef-414b-9d4b-fd327626422a admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:48.191554 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862ddbac-eaef-414b-9d4b-fd327626422a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:48.191928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862ddbac-eaef-414b-9d4b-fd327626422a admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:48.193082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862ddbac-eaef-414b-9d4b-fd327626422a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:48.196782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862ddbac-eaef-414b-9d4b-fd327626422a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:48.215674 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1063/2131] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:48 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:48.225349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136676c2-e627-46a4-a239-acb616381007 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:48.244189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136676c2-e627-46a4-a239-acb616381007 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:48.252664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c393425f-0a0b-4130-95cd-b8c3972182d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:48.253713 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c393425f-0a0b-4130-95cd-b8c3972182d8 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:48.254224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c393425f-0a0b-4130-95cd-b8c3972182d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:48.254671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c393425f-0a0b-4130-95cd-b8c3972182d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:48.255706 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c393425f-0a0b-4130-95cd-b8c3972182d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:48.258700 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c393425f-0a0b-4130-95cd-b8c3972182d8 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:48.280146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1069/2132] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:48 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:09:48.284696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862ddbac-eaef-414b-9d4b-fd327626422a admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:48.298254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862ddbac-eaef-414b-9d4b-fd327626422a admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:48.305502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86560971-8a5a-4d61-95c4-4ca3bcaf384b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:48.306279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86560971-8a5a-4d61-95c4-4ca3bcaf384b admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:48.306770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86560971-8a5a-4d61-95c4-4ca3bcaf384b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:48.307235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86560971-8a5a-4d61-95c4-4ca3bcaf384b admin admin] PATH_INFO: `/v3/projects/a42ab953bdf2412a97e41e971d859971/users/bbb4d22a094c45869374fce82c52d3b6/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:48.335586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86560971-8a5a-4d61-95c4-4ca3bcaf384b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a42ab953bdf2412a97e41e971d859971, user_id=bbb4d22a094c45869374fce82c52d3b6, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:48.338437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86560971-8a5a-4d61-95c4-4ca3bcaf384b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:48.354390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1064/2133] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:48 2022] PUT /identity/v3/projects/a42ab953bdf2412a97e41e971d859971/users/bbb4d22a094c45869374fce82c52d3b6/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:48.367578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860a2055-7188-44ba-9b8c-ca737dcbff81 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:48.367995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860a2055-7188-44ba-9b8c-ca737dcbff81 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:48.368358 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860a2055-7188-44ba-9b8c-ca737dcbff81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:48.403934 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-860a2055-7188-44ba-9b8c-ca737dcbff81 None None] MFA Rules not processed for user `bbb4d22a094c45869374fce82c52d3b6`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:48.405263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-860a2055-7188-44ba-9b8c-ca737dcbff81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:48.484181 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1070/2134] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:48 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:48.489023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86560971-8a5a-4d61-95c4-4ca3bcaf384b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:48.519484 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:48.589775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86560971-8a5a-4d61-95c4-4ca3bcaf384b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:48.597175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8a8743-edf3-4191-8e08-badd6d5e718f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:48.598020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8a8743-edf3-4191-8e08-badd6d5e718f admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:48.598370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8a8743-edf3-4191-8e08-badd6d5e718f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:48.598814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8a8743-edf3-4191-8e08-badd6d5e718f admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:48.599606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8a8743-edf3-4191-8e08-badd6d5e718f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:48.602406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8a8743-edf3-4191-8e08-badd6d5e718f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:48.656468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1065/2135] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:48 2022] POST /identity/v3/projects => generated 383 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:09:48.658361 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860a2055-7188-44ba-9b8c-ca737dcbff81 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:48.679148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860a2055-7188-44ba-9b8c-ca737dcbff81 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:48.741757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:48.770908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcb05ca-6e24-46f3-825c-e5205c4440c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:48.772150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcb05ca-6e24-46f3-825c-e5205c4440c3 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:48.772519 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcb05ca-6e24-46f3-825c-e5205c4440c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:48.772935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcb05ca-6e24-46f3-825c-e5205c4440c3 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:48.773874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcb05ca-6e24-46f3-825c-e5205c4440c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:48.775913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcb05ca-6e24-46f3-825c-e5205c4440c3 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:48.808059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1071/2136] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:48 2022] POST /identity/v3/users => generated 298 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:48.812951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8a8743-edf3-4191-8e08-badd6d5e718f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:48.824579 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8a8743-edf3-4191-8e08-badd6d5e718f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:48.829965 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2572574-748a-4949-b50e-218b00b6460d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:48.830886 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2572574-748a-4949-b50e-218b00b6460d admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:48.831276 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2572574-748a-4949-b50e-218b00b6460d admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:48.831655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2572574-748a-4949-b50e-218b00b6460d admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:48.832491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2572574-748a-4949-b50e-218b00b6460d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:48.834197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2572574-748a-4949-b50e-218b00b6460d admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:48.847233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1066/2137] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:48 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:48.851960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcb05ca-6e24-46f3-825c-e5205c4440c3 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:48.863639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcb05ca-6e24-46f3-825c-e5205c4440c3 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:48.869877 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002c6340-70f3-4530-a643-cd40848a16a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:48.870575 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002c6340-70f3-4530-a643-cd40848a16a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:48.870883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002c6340-70f3-4530-a643-cd40848a16a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:48.871168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002c6340-70f3-4530-a643-cd40848a16a1 admin admin] PATH_INFO: `/v3/projects/32c56170c7ec421196cc95704d1f136f/users/dbf0ad05e9884a2b9edb7e5fc40fd536/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:48.889615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002c6340-70f3-4530-a643-cd40848a16a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32c56170c7ec421196cc95704d1f136f, user_id=dbf0ad05e9884a2b9edb7e5fc40fd536, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:48.892829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002c6340-70f3-4530-a643-cd40848a16a1 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:48.912580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1072/2138] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:48 2022] PUT /identity/v3/projects/32c56170c7ec421196cc95704d1f136f/users/dbf0ad05e9884a2b9edb7e5fc40fd536/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:48.916666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2572574-748a-4949-b50e-218b00b6460d admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:48.928979 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2572574-748a-4949-b50e-218b00b6460d admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:48.998958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956547a9-c290-43c2-aa39-0a20341fc95b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:48.999771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956547a9-c290-43c2-aa39-0a20341fc95b admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:49.000108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956547a9-c290-43c2-aa39-0a20341fc95b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:49.000431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956547a9-c290-43c2-aa39-0a20341fc95b admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:49.001202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956547a9-c290-43c2-aa39-0a20341fc95b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:49.003212 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956547a9-c290-43c2-aa39-0a20341fc95b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:49.015648 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1067/2139] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:48 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:49.020348 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002c6340-70f3-4530-a643-cd40848a16a1 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:49.032321 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002c6340-70f3-4530-a643-cd40848a16a1 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:49.042419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749b22a5-4752-45e2-ae0e-e07ffa6aeb88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:49.043341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749b22a5-4752-45e2-ae0e-e07ffa6aeb88 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:49.043793 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749b22a5-4752-45e2-ae0e-e07ffa6aeb88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:49.044195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749b22a5-4752-45e2-ae0e-e07ffa6aeb88 admin admin] PATH_INFO: `/v3/projects/32c56170c7ec421196cc95704d1f136f/users/dbf0ad05e9884a2b9edb7e5fc40fd536/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:49.047212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749b22a5-4752-45e2-ae0e-e07ffa6aeb88 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32c56170c7ec421196cc95704d1f136f, user_id=dbf0ad05e9884a2b9edb7e5fc40fd536, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:49.050547 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749b22a5-4752-45e2-ae0e-e07ffa6aeb88 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:49.067500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1073/2140] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:49 2022] PUT /identity/v3/projects/32c56170c7ec421196cc95704d1f136f/users/dbf0ad05e9884a2b9edb7e5fc40fd536/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:49.083281 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08208d5-f8c2-40e6-9f5b-6f24a0b241e6 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:49.084368 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08208d5-f8c2-40e6-9f5b-6f24a0b241e6 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:49.084875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08208d5-f8c2-40e6-9f5b-6f24a0b241e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:49.122364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c08208d5-f8c2-40e6-9f5b-6f24a0b241e6 None None] MFA Rules not processed for user `dbf0ad05e9884a2b9edb7e5fc40fd536`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:49.123709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c08208d5-f8c2-40e6-9f5b-6f24a0b241e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:49.261703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1068/2141] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:49 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:49.271534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749b22a5-4752-45e2-ae0e-e07ffa6aeb88 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:49.274216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-749b22a5-4752-45e2-ae0e-e07ffa6aeb88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:49.356334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749b22a5-4752-45e2-ae0e-e07ffa6aeb88 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:49.361320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25558263-2c50-42e4-bf8d-c16dc9bb5cb6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:49.361964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25558263-2c50-42e4-bf8d-c16dc9bb5cb6 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:49.362235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25558263-2c50-42e4-bf8d-c16dc9bb5cb6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:49.362493 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25558263-2c50-42e4-bf8d-c16dc9bb5cb6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:49.448124 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25558263-2c50-42e4-bf8d-c16dc9bb5cb6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:49.449568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25558263-2c50-42e4-bf8d-c16dc9bb5cb6 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:49.459828 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1074/2142] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:49 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:09:49.607446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08208d5-f8c2-40e6-9f5b-6f24a0b241e6 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:49.610372 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c08208d5-f8c2-40e6-9f5b-6f24a0b241e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:49.624507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08208d5-f8c2-40e6-9f5b-6f24a0b241e6 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:49.628452 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7af3cbb-89ae-4547-8cd1-fe8937108a35 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:49.631305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7af3cbb-89ae-4547-8cd1-fe8937108a35 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:49.631305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7af3cbb-89ae-4547-8cd1-fe8937108a35 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:49.631305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7af3cbb-89ae-4547-8cd1-fe8937108a35 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:49.631846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b7af3cbb-89ae-4547-8cd1-fe8937108a35 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:49.651515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:49.730461 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7af3cbb-89ae-4547-8cd1-fe8937108a35 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:49.732544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7af3cbb-89ae-4547-8cd1-fe8937108a35 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:49.748015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1069/2143] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:49 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:50.201584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25558263-2c50-42e4-bf8d-c16dc9bb5cb6 service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:50.204503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-25558263-2c50-42e4-bf8d-c16dc9bb5cb6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:50.356996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:50.389245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25558263-2c50-42e4-bf8d-c16dc9bb5cb6 service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:50.396236 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:50.397289 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:50.397754 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:50.398125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] PATH_INFO: `/v3/users/48790f68a40444aa9e8b5a5f8454e8c6` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:50.399776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] RBAC: Authorizing `identity:delete_user(user_id=48790f68a40444aa9e8b5a5f8454e8c6)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:50.402643 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:50.462819 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '48790f68a40444aa9e8b5a5f8454e8c6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:50.463671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '48790f68a40444aa9e8b5a5f8454e8c6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:50.474335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '48790f68a40444aa9e8b5a5f8454e8c6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:50.493670 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '48790f68a40444aa9e8b5a5f8454e8c6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:50.502375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '48790f68a40444aa9e8b5a5f8454e8c6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:50.516753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1075/2144] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:50 2022] DELETE /identity/v3/users/48790f68a40444aa9e8b5a5f8454e8c6 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:50.523599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d86a3d-f0b9-409b-bf8d-f8c1e76846f2 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:50.523914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d86a3d-f0b9-409b-bf8d-f8c1e76846f2 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:50.524148 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d86a3d-f0b9-409b-bf8d-f8c1e76846f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:50.547335 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-86d86a3d-f0b9-409b-bf8d-f8c1e76846f2 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:50.548631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-86d86a3d-f0b9-409b-bf8d-f8c1e76846f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:50.635692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1070/2145] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:09:50.647080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:50.649496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:50.720186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efd5920-54a2-4fc2-86e1-d5b9f0e8fbea admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:50.724244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73120b9-8a4d-49c2-837e-840ae3005537 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:50.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73120b9-8a4d-49c2-837e-840ae3005537 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:50.725158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73120b9-8a4d-49c2-837e-840ae3005537 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:50.725429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73120b9-8a4d-49c2-837e-840ae3005537 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:50.735656 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73120b9-8a4d-49c2-837e-840ae3005537 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:50.737116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73120b9-8a4d-49c2-837e-840ae3005537 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:50.748003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1076/2146] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:51.055444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d86a3d-f0b9-409b-bf8d-f8c1e76846f2 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:51.056920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-86d86a3d-f0b9-409b-bf8d-f8c1e76846f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:51.070513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d86a3d-f0b9-409b-bf8d-f8c1e76846f2 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:51.079088 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:51.080532 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:51.080799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:51.088465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] PATH_INFO: `/v3/projects/02877c30378a4d4c978513ad644e6893` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:51.090371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=02877c30378a4d4c978513ad644e6893)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:51.092427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:51.197916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02877c30378a4d4c978513ad644e6893', 'request_id': 'req-2f185a6d-299d-4529-9e80-0a8d5f1637a5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:51.210426 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02877c30378a4d4c978513ad644e6893', 'request_id': 'req-2f185a6d-299d-4529-9e80-0a8d5f1637a5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:51.222023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02877c30378a4d4c978513ad644e6893', 'request_id': 'req-2f185a6d-299d-4529-9e80-0a8d5f1637a5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:51.243734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] The token cache is being invalidate because project 02877c30378a4d4c978513ad644e6893 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:51.244050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:51.244822 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:51.257913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1071/2147] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:51 2022] DELETE /identity/v3/projects/02877c30378a4d4c978513ad644e6893 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:51.263792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73120b9-8a4d-49c2-837e-840ae3005537 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:51.296600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73120b9-8a4d-49c2-837e-840ae3005537 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:51.424753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7b242e-ce44-41e2-8642-a64893a53ea1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:51.425648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b242e-ce44-41e2-8642-a64893a53ea1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:51.425959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b242e-ce44-41e2-8642-a64893a53ea1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:51.426334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b242e-ce44-41e2-8642-a64893a53ea1 admin admin] PATH_INFO: `/v3/users/7df159d06a774c6bac8c6a23799bbe06` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:51.428403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7b242e-ce44-41e2-8642-a64893a53ea1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7df159d06a774c6bac8c6a23799bbe06)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:51.430973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7b242e-ce44-41e2-8642-a64893a53ea1 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:51.446486 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:51.531288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8d7b242e-ce44-41e2-8642-a64893a53ea1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7df159d06a774c6bac8c6a23799bbe06'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:51.532188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8d7b242e-ce44-41e2-8642-a64893a53ea1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7df159d06a774c6bac8c6a23799bbe06'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:51.563741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8d7b242e-ce44-41e2-8642-a64893a53ea1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7df159d06a774c6bac8c6a23799bbe06'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:51.618857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8d7b242e-ce44-41e2-8642-a64893a53ea1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7df159d06a774c6bac8c6a23799bbe06'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:51.643451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8d7b242e-ce44-41e2-8642-a64893a53ea1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7df159d06a774c6bac8c6a23799bbe06'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:51.666505 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1077/2148] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:51 2022] DELETE /identity/v3/users/7df159d06a774c6bac8c6a23799bbe06 => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:52.130151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:52.131728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:52.153106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f185a6d-299d-4529-9e80-0a8d5f1637a5 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:52.212092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:52.249861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c3188b-305b-41c2-9ab7-ce3c43de7171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:52.251016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c3188b-305b-41c2-9ab7-ce3c43de7171 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:52.251560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c3188b-305b-41c2-9ab7-ce3c43de7171 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:52.252063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c3188b-305b-41c2-9ab7-ce3c43de7171 admin admin] PATH_INFO: `/v3/projects/97a6b783f1224584b214edbe1c5568f6` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:52.254303 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c3188b-305b-41c2-9ab7-ce3c43de7171 admin admin] RBAC: Authorizing `identity:delete_project(project_id=97a6b783f1224584b214edbe1c5568f6)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:52.257796 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c3188b-305b-41c2-9ab7-ce3c43de7171 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:52.379428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-66c3188b-305b-41c2-9ab7-ce3c43de7171 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '97a6b783f1224584b214edbe1c5568f6', 'request_id': 'req-66c3188b-305b-41c2-9ab7-ce3c43de7171'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:52.394075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-66c3188b-305b-41c2-9ab7-ce3c43de7171 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '97a6b783f1224584b214edbe1c5568f6', 'request_id': 'req-66c3188b-305b-41c2-9ab7-ce3c43de7171'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:52.415243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-66c3188b-305b-41c2-9ab7-ce3c43de7171 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '97a6b783f1224584b214edbe1c5568f6', 'request_id': 'req-66c3188b-305b-41c2-9ab7-ce3c43de7171'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:52.443970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-66c3188b-305b-41c2-9ab7-ce3c43de7171 admin admin] The token cache is being invalidate because project 97a6b783f1224584b214edbe1c5568f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:52.444530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-66c3188b-305b-41c2-9ab7-ce3c43de7171 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:52.446083 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-66c3188b-305b-41c2-9ab7-ce3c43de7171 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:52.449101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1072/2149] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:52 2022] DELETE /identity/v3/projects/97a6b783f1224584b214edbe1c5568f6 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:52.455296 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f8620d-6716-426b-9123-2892a8b87ad6 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:52.456054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f8620d-6716-426b-9123-2892a8b87ad6 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:52.456602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f8620d-6716-426b-9123-2892a8b87ad6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:52.533818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-24f8620d-6716-426b-9123-2892a8b87ad6 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:52.535917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-24f8620d-6716-426b-9123-2892a8b87ad6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:52.721502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1078/2150] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:52.737003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf43a44e-ac29-4ddc-927c-cc96de20e7d8 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:52.739540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf43a44e-ac29-4ddc-927c-cc96de20e7d8 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:52.740519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf43a44e-ac29-4ddc-927c-cc96de20e7d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:52.783406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-bf43a44e-ac29-4ddc-927c-cc96de20e7d8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:52.785813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-bf43a44e-ac29-4ddc-927c-cc96de20e7d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:52.801475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1073/2151] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:09:52.806447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f8620d-6716-426b-9123-2892a8b87ad6 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:52.822389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f8620d-6716-426b-9123-2892a8b87ad6 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:52.832122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d33011a-b622-462c-80b9-a3f375ee0ec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:52.833870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d33011a-b622-462c-80b9-a3f375ee0ec4 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:52.834628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d33011a-b622-462c-80b9-a3f375ee0ec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:52.835174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d33011a-b622-462c-80b9-a3f375ee0ec4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:52.836544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d33011a-b622-462c-80b9-a3f375ee0ec4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:52.837163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d33011a-b622-462c-80b9-a3f375ee0ec4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:52.841170 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d33011a-b622-462c-80b9-a3f375ee0ec4 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:52.866681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1079/2152] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:09:52.876757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf43a44e-ac29-4ddc-927c-cc96de20e7d8 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:52.889975 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf43a44e-ac29-4ddc-927c-cc96de20e7d8 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:52.896864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614dfe71-c8a7-43db-88c3-bb4d81d6a2ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:52.897544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614dfe71-c8a7-43db-88c3-bb4d81d6a2ec admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:52.897981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614dfe71-c8a7-43db-88c3-bb4d81d6a2ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:52.900533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614dfe71-c8a7-43db-88c3-bb4d81d6a2ec admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:52.901391 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614dfe71-c8a7-43db-88c3-bb4d81d6a2ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:52.904156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614dfe71-c8a7-43db-88c3-bb4d81d6a2ec admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:52.957951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1074/2153] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:52 2022] POST /identity/v3/projects => generated 361 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:52.964833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d33011a-b622-462c-80b9-a3f375ee0ec4 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:52.983549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d33011a-b622-462c-80b9-a3f375ee0ec4 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:53.093701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606f09bf-7326-4476-a46d-6ec9558b1cb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:53.094734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606f09bf-7326-4476-a46d-6ec9558b1cb2 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:53.095270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606f09bf-7326-4476-a46d-6ec9558b1cb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:53.095771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606f09bf-7326-4476-a46d-6ec9558b1cb2 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:53.096928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606f09bf-7326-4476-a46d-6ec9558b1cb2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:53.099194 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606f09bf-7326-4476-a46d-6ec9558b1cb2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:53.135464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1080/2154] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:52 2022] POST /identity/v3/users => generated 281 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:09:53.138054 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614dfe71-c8a7-43db-88c3-bb4d81d6a2ec admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:53.160142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614dfe71-c8a7-43db-88c3-bb4d81d6a2ec admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:53.169795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7f3d39-3550-455d-9e0b-2267b5d0b537 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:53.170828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7f3d39-3550-455d-9e0b-2267b5d0b537 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:53.171452 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7f3d39-3550-455d-9e0b-2267b5d0b537 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:53.171955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7f3d39-3550-455d-9e0b-2267b5d0b537 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:53.173553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7f3d39-3550-455d-9e0b-2267b5d0b537 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:53.177056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7f3d39-3550-455d-9e0b-2267b5d0b537 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:53.236727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1075/2155] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:53 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:53.238914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606f09bf-7326-4476-a46d-6ec9558b1cb2 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:53.254578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:53.267091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606f09bf-7326-4476-a46d-6ec9558b1cb2 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:53.283452 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65972345-f745-4e54-b9d2-0d5f12e2615b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:53.285465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65972345-f745-4e54-b9d2-0d5f12e2615b admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:53.285465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65972345-f745-4e54-b9d2-0d5f12e2615b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:53.285789 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65972345-f745-4e54-b9d2-0d5f12e2615b admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:53.287328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65972345-f745-4e54-b9d2-0d5f12e2615b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:53.290484 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65972345-f745-4e54-b9d2-0d5f12e2615b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:53.323206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1081/2156] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:53 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:53.324132 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7f3d39-3550-455d-9e0b-2267b5d0b537 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:53.338229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7f3d39-3550-455d-9e0b-2267b5d0b537 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:53.345000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3891c6-9609-42a3-9d75-1dfbe0be3710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:53.346075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3891c6-9609-42a3-9d75-1dfbe0be3710 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:53.346575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3891c6-9609-42a3-9d75-1dfbe0be3710 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:53.346843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3891c6-9609-42a3-9d75-1dfbe0be3710 admin admin] PATH_INFO: `/v3/projects/fd92dce9889f4d869548031bec66dc6e/users/3e604473e7364d2dad46570037904100/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:53.366498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3891c6-9609-42a3-9d75-1dfbe0be3710 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd92dce9889f4d869548031bec66dc6e, user_id=3e604473e7364d2dad46570037904100, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:53.369743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3891c6-9609-42a3-9d75-1dfbe0be3710 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:53.424695 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1076/2157] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:53 2022] PUT /identity/v3/projects/fd92dce9889f4d869548031bec66dc6e/users/3e604473e7364d2dad46570037904100/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:53.446611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e46b55-7c7f-46fe-949d-9bb7fb147e45 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:53.447067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e46b55-7c7f-46fe-949d-9bb7fb147e45 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:53.447586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e46b55-7c7f-46fe-949d-9bb7fb147e45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:53.486487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-70e46b55-7c7f-46fe-949d-9bb7fb147e45 None None] MFA Rules not processed for user `3e604473e7364d2dad46570037904100`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:53.487788 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-70e46b55-7c7f-46fe-949d-9bb7fb147e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:53.595006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1082/2158] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:53 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:53.863907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3891c6-9609-42a3-9d75-1dfbe0be3710 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:53.866660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2e3891c6-9609-42a3-9d75-1dfbe0be3710 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:53.972882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3891c6-9609-42a3-9d75-1dfbe0be3710 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:53.981310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840ac44f-29c4-4d63-961f-e401c0235384 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:53.982632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840ac44f-29c4-4d63-961f-e401c0235384 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:53.983143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840ac44f-29c4-4d63-961f-e401c0235384 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:53.984370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840ac44f-29c4-4d63-961f-e401c0235384 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:53.996491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840ac44f-29c4-4d63-961f-e401c0235384 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:53.998184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840ac44f-29c4-4d63-961f-e401c0235384 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:54.019707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1077/2159] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:53 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:54.037747 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e46b55-7c7f-46fe-949d-9bb7fb147e45 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:54.155981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e46b55-7c7f-46fe-949d-9bb7fb147e45 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:54.162004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:54.162674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:54.162952 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:54.163463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] PATH_INFO: `/v3/users/3e604473e7364d2dad46570037904100` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:54.164923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] RBAC: Authorizing `identity:delete_user(user_id=3e604473e7364d2dad46570037904100)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:54.167414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:54.245089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3e604473e7364d2dad46570037904100'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:54.246007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3e604473e7364d2dad46570037904100'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:54.257378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3e604473e7364d2dad46570037904100'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:54.272846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3e604473e7364d2dad46570037904100'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:54.282670 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3e604473e7364d2dad46570037904100'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:54.291819 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1083/2160] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:54 2022] DELETE /identity/v3/users/3e604473e7364d2dad46570037904100 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:54.299266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0adf0a-f894-4f7c-a3b1-b4b6c421991a None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:54.300117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0adf0a-f894-4f7c-a3b1-b4b6c421991a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:54.300117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0adf0a-f894-4f7c-a3b1-b4b6c421991a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:54.317484 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:54.345223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ed0adf0a-f894-4f7c-a3b1-b4b6c421991a None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:54.347603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ed0adf0a-f894-4f7c-a3b1-b4b6c421991a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:54.435213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1078/2161] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:09:54.448624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:54.450930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:54.460602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:54.542253 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff97da78-4dab-4ba5-85fe-7ee89248685d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:54.550810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e643405-68ae-49d3-8137-72f13091fa4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:54.552803 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e643405-68ae-49d3-8137-72f13091fa4f service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:54.553270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e643405-68ae-49d3-8137-72f13091fa4f service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:54.553693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e643405-68ae-49d3-8137-72f13091fa4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:54.577428 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e643405-68ae-49d3-8137-72f13091fa4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:54.581039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e643405-68ae-49d3-8137-72f13091fa4f service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:54.600857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1084/2162] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:55.069699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0adf0a-f894-4f7c-a3b1-b4b6c421991a None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:55.071252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ed0adf0a-f894-4f7c-a3b1-b4b6c421991a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:55.100008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0adf0a-f894-4f7c-a3b1-b4b6c421991a None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:55.107688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f9bbb8-8259-4e9d-8bcf-16560a673115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:55.109350 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f9bbb8-8259-4e9d-8bcf-16560a673115 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:55.110199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f9bbb8-8259-4e9d-8bcf-16560a673115 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:55.110828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f9bbb8-8259-4e9d-8bcf-16560a673115 admin admin] PATH_INFO: `/v3/projects/fd92dce9889f4d869548031bec66dc6e` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:55.113052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f9bbb8-8259-4e9d-8bcf-16560a673115 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd92dce9889f4d869548031bec66dc6e)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:55.115118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f9bbb8-8259-4e9d-8bcf-16560a673115 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:55.138276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e643405-68ae-49d3-8137-72f13091fa4f service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:55.140940 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0e643405-68ae-49d3-8137-72f13091fa4f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:55.204517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-68f9bbb8-8259-4e9d-8bcf-16560a673115 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd92dce9889f4d869548031bec66dc6e', 'request_id': 'req-68f9bbb8-8259-4e9d-8bcf-16560a673115'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:55.219439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-68f9bbb8-8259-4e9d-8bcf-16560a673115 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd92dce9889f4d869548031bec66dc6e', 'request_id': 'req-68f9bbb8-8259-4e9d-8bcf-16560a673115'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:55.228437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-68f9bbb8-8259-4e9d-8bcf-16560a673115 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd92dce9889f4d869548031bec66dc6e', 'request_id': 'req-68f9bbb8-8259-4e9d-8bcf-16560a673115'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:55.230925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e643405-68ae-49d3-8137-72f13091fa4f service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:55.243847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-68f9bbb8-8259-4e9d-8bcf-16560a673115 admin admin] The token cache is being invalidate because project fd92dce9889f4d869548031bec66dc6e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:55.244346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-68f9bbb8-8259-4e9d-8bcf-16560a673115 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:55.245134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-68f9bbb8-8259-4e9d-8bcf-16560a673115 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:55.250794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1079/2163] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:55 2022] DELETE /identity/v3/projects/fd92dce9889f4d869548031bec66dc6e => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:55.253102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a1aedc-4b18-4693-b2ac-c14affa19c9d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:55.253557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a1aedc-4b18-4693-b2ac-c14affa19c9d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:55.254933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a1aedc-4b18-4693-b2ac-c14affa19c9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:55.302851 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-87a1aedc-4b18-4693-b2ac-c14affa19c9d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:55.304276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b150c9-de31-4130-8b40-4a0cbef9a43a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:55.304650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-87a1aedc-4b18-4693-b2ac-c14affa19c9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:55.304930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b150c9-de31-4130-8b40-4a0cbef9a43a service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:55.305236 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b150c9-de31-4130-8b40-4a0cbef9a43a service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:55.305514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b150c9-de31-4130-8b40-4a0cbef9a43a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:55.308024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d4b150c9-de31-4130-8b40-4a0cbef9a43a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:55.423699 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b150c9-de31-4130-8b40-4a0cbef9a43a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:55.426639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b150c9-de31-4130-8b40-4a0cbef9a43a service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:55.441962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1080/2164] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:55.446697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944b94b0-3457-414a-961a-f25f9f998544 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:55.447693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944b94b0-3457-414a-961a-f25f9f998544 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:55.448105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944b94b0-3457-414a-961a-f25f9f998544 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:55.468759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1085/2165] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:55 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:09:55.470917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b150c9-de31-4130-8b40-4a0cbef9a43a service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:55.473549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d4b150c9-de31-4130-8b40-4a0cbef9a43a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:55.476283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-944b94b0-3457-414a-961a-f25f9f998544 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:55.477955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-944b94b0-3457-414a-961a-f25f9f998544 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:55.494466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1081/2166] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:55.496839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944b94b0-3457-414a-961a-f25f9f998544 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:55.508991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944b94b0-3457-414a-961a-f25f9f998544 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:55.515074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3879e1-633e-4e69-b258-e116cea9b40c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:55.515854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3879e1-633e-4e69-b258-e116cea9b40c admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:55.516096 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3879e1-633e-4e69-b258-e116cea9b40c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:55.516337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3879e1-633e-4e69-b258-e116cea9b40c admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:55.517238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3879e1-633e-4e69-b258-e116cea9b40c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:09:55.517680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3879e1-633e-4e69-b258-e116cea9b40c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:55.519257 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3879e1-633e-4e69-b258-e116cea9b40c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:55.521284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b150c9-de31-4130-8b40-4a0cbef9a43a service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:55.534438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cd7992-6389-4396-81a3-3fd048ca4032 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:55.535212 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cd7992-6389-4396-81a3-3fd048ca4032 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:55.535759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cd7992-6389-4396-81a3-3fd048ca4032 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:55.536043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1082/2167] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:09:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:09:55.536347 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cd7992-6389-4396-81a3-3fd048ca4032 admin admin] PATH_INFO: `/v3/users/e2c4ac1fe934430dae8418ddbc20d25a` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:55.537579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74cd7992-6389-4396-81a3-3fd048ca4032 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e2c4ac1fe934430dae8418ddbc20d25a)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:55.539773 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74cd7992-6389-4396-81a3-3fd048ca4032 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:55.540483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3879e1-633e-4e69-b258-e116cea9b40c admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:55.551422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:55.564843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3879e1-633e-4e69-b258-e116cea9b40c admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:55.581943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79087f3-793e-4cbd-8ad7-856b768dd6ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:55.582738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79087f3-793e-4cbd-8ad7-856b768dd6ca admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:55.583068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79087f3-793e-4cbd-8ad7-856b768dd6ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:55.584008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79087f3-793e-4cbd-8ad7-856b768dd6ca admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:55.585191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79087f3-793e-4cbd-8ad7-856b768dd6ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:55.587621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79087f3-793e-4cbd-8ad7-856b768dd6ca admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:55.596430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:55.653716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1083/2168] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:09:55 2022] POST /identity/v3/projects => generated 375 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:09:55.657433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79087f3-793e-4cbd-8ad7-856b768dd6ca admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:55.663938 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-74cd7992-6389-4396-81a3-3fd048ca4032 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e2c4ac1fe934430dae8418ddbc20d25a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:55.665544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-74cd7992-6389-4396-81a3-3fd048ca4032 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e2c4ac1fe934430dae8418ddbc20d25a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:55.681808 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-74cd7992-6389-4396-81a3-3fd048ca4032 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e2c4ac1fe934430dae8418ddbc20d25a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:55.732318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-74cd7992-6389-4396-81a3-3fd048ca4032 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e2c4ac1fe934430dae8418ddbc20d25a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:55.762654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-74cd7992-6389-4396-81a3-3fd048ca4032 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e2c4ac1fe934430dae8418ddbc20d25a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:55.775425 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1086/2169] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:55 2022] DELETE /identity/v3/users/e2c4ac1fe934430dae8418ddbc20d25a => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:55.777144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd200b5-1999-49bc-8a96-96c87d369641 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:55.777437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd200b5-1999-49bc-8a96-96c87d369641 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:55.777714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd200b5-1999-49bc-8a96-96c87d369641 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:55.824722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-0dd200b5-1999-49bc-8a96-96c87d369641 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:55.826030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0dd200b5-1999-49bc-8a96-96c87d369641 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:55.849502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79087f3-793e-4cbd-8ad7-856b768dd6ca admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:55.933172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1087/2170] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:55.940958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd200b5-1999-49bc-8a96-96c87d369641 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:55.945487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0dd200b5-1999-49bc-8a96-96c87d369641 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:55.986928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38058e78-7b54-4de4-a2f2-c1379d0a6df8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:55.988582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38058e78-7b54-4de4-a2f2-c1379d0a6df8 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:55.991617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38058e78-7b54-4de4-a2f2-c1379d0a6df8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:55.992098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38058e78-7b54-4de4-a2f2-c1379d0a6df8 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:56.013708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38058e78-7b54-4de4-a2f2-c1379d0a6df8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:56.023429 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38058e78-7b54-4de4-a2f2-c1379d0a6df8 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:56.056878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd200b5-1999-49bc-8a96-96c87d369641 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:56.074118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1084/2171] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:09:55 2022] POST /identity/v3/users => generated 288 bytes in 419 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:09:56.078201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aebee6b-c199-429c-9b4c-292633b7127f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:56.079300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38058e78-7b54-4de4-a2f2-c1379d0a6df8 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:56.081298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-38058e78-7b54-4de4-a2f2-c1379d0a6df8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:56.081883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aebee6b-c199-429c-9b4c-292633b7127f service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:56.085637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aebee6b-c199-429c-9b4c-292633b7127f service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:56.086280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aebee6b-c199-429c-9b4c-292633b7127f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:56.096659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38058e78-7b54-4de4-a2f2-c1379d0a6df8 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:56.103787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb200e3-3ced-4ba2-8d3d-3f73287a1891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:56.103787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb200e3-3ced-4ba2-8d3d-3f73287a1891 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:56.104269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb200e3-3ced-4ba2-8d3d-3f73287a1891 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:56.104511 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb200e3-3ced-4ba2-8d3d-3f73287a1891 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:56.111399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb200e3-3ced-4ba2-8d3d-3f73287a1891 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:56.111399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb200e3-3ced-4ba2-8d3d-3f73287a1891 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:56.140778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aebee6b-c199-429c-9b4c-292633b7127f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:56.144048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aebee6b-c199-429c-9b4c-292633b7127f service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:56.148175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1085/2172] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:56 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:56.148686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb200e3-3ced-4ba2-8d3d-3f73287a1891 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:56.162864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb200e3-3ced-4ba2-8d3d-3f73287a1891 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:56.166284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1088/2173] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:56.167441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aebee6b-c199-429c-9b4c-292633b7127f service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:56.173980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2aebee6b-c199-429c-9b4c-292633b7127f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:56.183072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f3cc95-7faa-4c2e-af8e-9c246af5f8ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:56.186020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3cc95-7faa-4c2e-af8e-9c246af5f8ac admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:56.188937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3cc95-7faa-4c2e-af8e-9c246af5f8ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:56.189421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3cc95-7faa-4c2e-af8e-9c246af5f8ac admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:56.190692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f3cc95-7faa-4c2e-af8e-9c246af5f8ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:56.193680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f3cc95-7faa-4c2e-af8e-9c246af5f8ac admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:56.204000 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aebee6b-c199-429c-9b4c-292633b7127f service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:56.218518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc70dc7-5447-4d1d-9fcb-59eae1998408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:56.220017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc70dc7-5447-4d1d-9fcb-59eae1998408 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:56.220572 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc70dc7-5447-4d1d-9fcb-59eae1998408 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:56.221114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc70dc7-5447-4d1d-9fcb-59eae1998408 admin admin] PATH_INFO: `/v3/users/6d800f7c273f4c9f8874f84231e8f696` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:56.224627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc70dc7-5447-4d1d-9fcb-59eae1998408 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d800f7c273f4c9f8874f84231e8f696)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:56.229962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc70dc7-5447-4d1d-9fcb-59eae1998408 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:56.239421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1086/2174] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:09:56 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:09:56.243844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f3cc95-7faa-4c2e-af8e-9c246af5f8ac admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:56.269927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f3cc95-7faa-4c2e-af8e-9c246af5f8ac admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:56.274430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:56.280126 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad63b43d-7f77-47b9-9100-5a99fed1f340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:56.282133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad63b43d-7f77-47b9-9100-5a99fed1f340 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:56.282865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad63b43d-7f77-47b9-9100-5a99fed1f340 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:56.283626 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad63b43d-7f77-47b9-9100-5a99fed1f340 admin admin] PATH_INFO: `/v3/projects/e4f21990c9ae4a2a9b9ca2c5680d9a1b/users/6e733490ff584013956a16c2611008e3/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:56.316458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad63b43d-7f77-47b9-9100-5a99fed1f340 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4f21990c9ae4a2a9b9ca2c5680d9a1b, user_id=6e733490ff584013956a16c2611008e3, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:56.319199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2bc70dc7-5447-4d1d-9fcb-59eae1998408 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d800f7c273f4c9f8874f84231e8f696'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:56.319746 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad63b43d-7f77-47b9-9100-5a99fed1f340 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:56.320708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2bc70dc7-5447-4d1d-9fcb-59eae1998408 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d800f7c273f4c9f8874f84231e8f696'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:56.335431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2bc70dc7-5447-4d1d-9fcb-59eae1998408 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d800f7c273f4c9f8874f84231e8f696'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:56.338059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1087/2175] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:09:56 2022] PUT /identity/v3/projects/e4f21990c9ae4a2a9b9ca2c5680d9a1b/users/6e733490ff584013956a16c2611008e3/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:56.354041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2bc70dc7-5447-4d1d-9fcb-59eae1998408 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d800f7c273f4c9f8874f84231e8f696'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:56.358324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d757489-5548-4299-b3cf-b9b2b6539c32 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:56.358659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d757489-5548-4299-b3cf-b9b2b6539c32 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:56.359151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d757489-5548-4299-b3cf-b9b2b6539c32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:56.363813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2bc70dc7-5447-4d1d-9fcb-59eae1998408 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d800f7c273f4c9f8874f84231e8f696'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:56.370806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1089/2176] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:56 2022] DELETE /identity/v3/users/6d800f7c273f4c9f8874f84231e8f696 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:56.376386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d176fdc-50ea-42c1-a4c4-018596c9dfb6 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:56.376812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d176fdc-50ea-42c1-a4c4-018596c9dfb6 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:56.377183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d176fdc-50ea-42c1-a4c4-018596c9dfb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:56.401057 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-7d176fdc-50ea-42c1-a4c4-018596c9dfb6 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:56.402963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7d176fdc-50ea-42c1-a4c4-018596c9dfb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:56.408954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-2d757489-5548-4299-b3cf-b9b2b6539c32 None None] MFA Rules not processed for user `6e733490ff584013956a16c2611008e3`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:56.410885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2d757489-5548-4299-b3cf-b9b2b6539c32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:56.466961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:56.521736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1090/2177] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:09:56.534556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d176fdc-50ea-42c1-a4c4-018596c9dfb6 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:56.538223 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7d176fdc-50ea-42c1-a4c4-018596c9dfb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:56.575197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1088/2178] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:56 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:09:56.591533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d757489-5548-4299-b3cf-b9b2b6539c32 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:56.594168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2d757489-5548-4299-b3cf-b9b2b6539c32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:56.671293 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d176fdc-50ea-42c1-a4c4-018596c9dfb6 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:56.679763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2216fce-d979-42d0-baa1-59ba85b7ac1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:56.680873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2216fce-d979-42d0-baa1-59ba85b7ac1f service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:56.681159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2216fce-d979-42d0-baa1-59ba85b7ac1f service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:56.681430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2216fce-d979-42d0-baa1-59ba85b7ac1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:56.694976 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d757489-5548-4299-b3cf-b9b2b6539c32 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:56.701105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead9a33-e90b-4a29-8f95-8d14d56a016b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:56.701784 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2216fce-d979-42d0-baa1-59ba85b7ac1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:56.702099 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead9a33-e90b-4a29-8f95-8d14d56a016b service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:56.702581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead9a33-e90b-4a29-8f95-8d14d56a016b service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:56.702987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead9a33-e90b-4a29-8f95-8d14d56a016b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:56.704383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2216fce-d979-42d0-baa1-59ba85b7ac1f service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:56.722001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aead9a33-e90b-4a29-8f95-8d14d56a016b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:56.725030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aead9a33-e90b-4a29-8f95-8d14d56a016b service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:56.727132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1091/2179] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:56.728631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2216fce-d979-42d0-baa1-59ba85b7ac1f service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:56.731118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f2216fce-d979-42d0-baa1-59ba85b7ac1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:56.743733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1089/2180] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:09:56 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:09:56.759682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2216fce-d979-42d0-baa1-59ba85b7ac1f service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:56.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:56.767648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:56.768124 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:56.768585 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] PATH_INFO: `/v3/projects/14a1550c1fa94b6985a8ae59d35b8ec8` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:56.770181 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] RBAC: Authorizing `identity:delete_project(project_id=14a1550c1fa94b6985a8ae59d35b8ec8)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:56.773093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:56.838114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:56.853169 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14a1550c1fa94b6985a8ae59d35b8ec8', 'request_id': 'req-5b00805a-2f81-4e5d-8f76-591c7f4f4602'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:56.867424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14a1550c1fa94b6985a8ae59d35b8ec8', 'request_id': 'req-5b00805a-2f81-4e5d-8f76-591c7f4f4602'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:56.882595 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14a1550c1fa94b6985a8ae59d35b8ec8', 'request_id': 'req-5b00805a-2f81-4e5d-8f76-591c7f4f4602'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:56.907779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] The token cache is being invalidate because project 14a1550c1fa94b6985a8ae59d35b8ec8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:56.907779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:56.908975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:56.911673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1092/2181] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:56 2022] DELETE /identity/v3/projects/14a1550c1fa94b6985a8ae59d35b8ec8 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:56.916541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead9a33-e90b-4a29-8f95-8d14d56a016b service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:56.932457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead9a33-e90b-4a29-8f95-8d14d56a016b service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:57.077488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:57.078777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:57.080706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:57.081475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] PATH_INFO: `/v3/users/6d4bf25fd67749209f8905737c11810a` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:57.085079 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d4bf25fd67749209f8905737c11810a)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:57.089500 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:57.230377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d4bf25fd67749209f8905737c11810a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:57.246940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d4bf25fd67749209f8905737c11810a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:57.248654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d4bf25fd67749209f8905737c11810a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:57.290463 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:57.296443 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d4bf25fd67749209f8905737c11810a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:57.300522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d4bf25fd67749209f8905737c11810a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:57.338301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1090/2182] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:56 2022] DELETE /identity/v3/users/6d4bf25fd67749209f8905737c11810a => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:57.425653 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:57.427457 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:57.450482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b00805a-2f81-4e5d-8f76-591c7f4f4602 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:57.534159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:57.534819 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:57.535133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:57.535432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] PATH_INFO: `/v3/projects/1ef4c28d3fc94ed6960965e1f3f2d2b8` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:57.536861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ef4c28d3fc94ed6960965e1f3f2d2b8)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:57.538947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:57.648115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ef4c28d3fc94ed6960965e1f3f2d2b8', 'request_id': 'req-9f5801d1-a03f-49ce-b07b-f28e90969498'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:57.672003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ef4c28d3fc94ed6960965e1f3f2d2b8', 'request_id': 'req-9f5801d1-a03f-49ce-b07b-f28e90969498'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:57.689130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ef4c28d3fc94ed6960965e1f3f2d2b8', 'request_id': 'req-9f5801d1-a03f-49ce-b07b-f28e90969498'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:57.720991 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] The token cache is being invalidate because project 1ef4c28d3fc94ed6960965e1f3f2d2b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:57.721877 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:57.723300 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:57.726478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1093/2183] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:57 2022] DELETE /identity/v3/projects/1ef4c28d3fc94ed6960965e1f3f2d2b8 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:57.732707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:57.735907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:57.779330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed4d95a-031b-44d2-a8e3-2e78a983826d admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:57.935933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:57.939547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:57.939874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:57.940159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] PATH_INFO: `/v3/users/b4b8449f572b40d3a70324fe9f711a08` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:57.944441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] RBAC: Authorizing `identity:delete_user(user_id=b4b8449f572b40d3a70324fe9f711a08)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:57.947881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:58.060071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b4b8449f572b40d3a70324fe9f711a08'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:58.084996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b4b8449f572b40d3a70324fe9f711a08'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:58.085124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b4b8449f572b40d3a70324fe9f711a08'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:58.106795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b4b8449f572b40d3a70324fe9f711a08'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:58.108907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b4b8449f572b40d3a70324fe9f711a08'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:58.140949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1091/2184] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:57 2022] DELETE /identity/v3/users/b4b8449f572b40d3a70324fe9f711a08 => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:58.692484 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:58.694251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:58.737752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5801d1-a03f-49ce-b07b-f28e90969498 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:58.907639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:58.959741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:58.961673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:58.962464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:58.964059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] PATH_INFO: `/v3/projects/784de17a2e9040fe8aba15604cdcc935` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:58.967128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] RBAC: Authorizing `identity:delete_project(project_id=784de17a2e9040fe8aba15604cdcc935)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:58.971563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:59.075598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '784de17a2e9040fe8aba15604cdcc935', 'request_id': 'req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.083252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '784de17a2e9040fe8aba15604cdcc935', 'request_id': 'req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.094262 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '784de17a2e9040fe8aba15604cdcc935', 'request_id': 'req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.109155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] The token cache is being invalidate because project 784de17a2e9040fe8aba15604cdcc935 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:59.109507 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.110372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.116637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1094/2185] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:58 2022] DELETE /identity/v3/projects/784de17a2e9040fe8aba15604cdcc935 => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:59.118507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:59.121221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:59.156713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4954a6-e7d6-4e94-97ab-6b5520175a9e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:59.327307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f9b100-4975-4b90-899c-77fd3074a766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:59.330310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f9b100-4975-4b90-899c-77fd3074a766 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:59.330914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f9b100-4975-4b90-899c-77fd3074a766 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:59.331610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f9b100-4975-4b90-899c-77fd3074a766 admin admin] PATH_INFO: `/v3/users/65e17774b06a480f90d4c627ca8f587f` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:59.334331 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f9b100-4975-4b90-899c-77fd3074a766 admin admin] RBAC: Authorizing `identity:delete_user(user_id=65e17774b06a480f90d4c627ca8f587f)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:59.340773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f9b100-4975-4b90-899c-77fd3074a766 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:59.374169 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:09:59.433160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-66f9b100-4975-4b90-899c-77fd3074a766 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65e17774b06a480f90d4c627ca8f587f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.445001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-66f9b100-4975-4b90-899c-77fd3074a766 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65e17774b06a480f90d4c627ca8f587f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.445914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-66f9b100-4975-4b90-899c-77fd3074a766 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65e17774b06a480f90d4c627ca8f587f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.463405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-66f9b100-4975-4b90-899c-77fd3074a766 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65e17774b06a480f90d4c627ca8f587f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.464675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-66f9b100-4975-4b90-899c-77fd3074a766 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65e17774b06a480f90d4c627ca8f587f'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.490263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1092/2186] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:09:59 2022] DELETE /identity/v3/users/65e17774b06a480f90d4c627ca8f587f => generated 0 bytes in 373 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:09:59.551640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:09:59.553294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:09:59.577821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e4c1ca-3f5d-429e-ab6d-a7a3a2d76215 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:09:59.707346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12effc8a-c497-44d3-8a4c-fecd1bd0e708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:09:59.708434 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12effc8a-c497-44d3-8a4c-fecd1bd0e708 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:59.708434 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12effc8a-c497-44d3-8a4c-fecd1bd0e708 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:59.708728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12effc8a-c497-44d3-8a4c-fecd1bd0e708 admin admin] PATH_INFO: `/v3/projects/5422d71676324b2e930a9e8d5f3a71cb` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:59.709938 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12effc8a-c497-44d3-8a4c-fecd1bd0e708 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5422d71676324b2e930a9e8d5f3a71cb)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:09:59.712156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12effc8a-c497-44d3-8a4c-fecd1bd0e708 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:09:59.817514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-12effc8a-c497-44d3-8a4c-fecd1bd0e708 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5422d71676324b2e930a9e8d5f3a71cb', 'request_id': 'req-12effc8a-c497-44d3-8a4c-fecd1bd0e708'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.826314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-12effc8a-c497-44d3-8a4c-fecd1bd0e708 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5422d71676324b2e930a9e8d5f3a71cb', 'request_id': 'req-12effc8a-c497-44d3-8a4c-fecd1bd0e708'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.841188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-12effc8a-c497-44d3-8a4c-fecd1bd0e708 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5422d71676324b2e930a9e8d5f3a71cb', 'request_id': 'req-12effc8a-c497-44d3-8a4c-fecd1bd0e708'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.860571 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-12effc8a-c497-44d3-8a4c-fecd1bd0e708 admin admin] The token cache is being invalidate because project 5422d71676324b2e930a9e8d5f3a71cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:09:59.862673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-12effc8a-c497-44d3-8a4c-fecd1bd0e708 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.863726 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-12effc8a-c497-44d3-8a4c-fecd1bd0e708 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:09:59.866137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1095/2187] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:09:59 2022] DELETE /identity/v3/projects/5422d71676324b2e930a9e8d5f3a71cb => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:09:59.873467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec7608e-1b07-4785-933a-d2fedb60b406 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:09:59.874424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec7608e-1b07-4785-933a-d2fedb60b406 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:09:59.874906 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec7608e-1b07-4785-933a-d2fedb60b406 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:09:59.904311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-fec7608e-1b07-4785-933a-d2fedb60b406 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:09:59.906596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fec7608e-1b07-4785-933a-d2fedb60b406 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:00.049908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1093/2188] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:09:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:00.060135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d1ce8c-8143-405b-b095-d754313ce533 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:00.060638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d1ce8c-8143-405b-b095-d754313ce533 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:00.060928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d1ce8c-8143-405b-b095-d754313ce533 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:00.090584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-53d1ce8c-8143-405b-b095-d754313ce533 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:00.092375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-53d1ce8c-8143-405b-b095-d754313ce533 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:00.103189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1096/2189] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:00.108163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec7608e-1b07-4785-933a-d2fedb60b406 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:00.122638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec7608e-1b07-4785-933a-d2fedb60b406 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:00.133447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8297e2de-c616-4068-8d68-24398bba81af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:00.134694 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8297e2de-c616-4068-8d68-24398bba81af admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:00.135015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8297e2de-c616-4068-8d68-24398bba81af admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:00.135545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8297e2de-c616-4068-8d68-24398bba81af admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:00.136860 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8297e2de-c616-4068-8d68-24398bba81af admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:10:00.137776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8297e2de-c616-4068-8d68-24398bba81af admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:00.141774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8297e2de-c616-4068-8d68-24398bba81af admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:00.163004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1094/2190] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:10:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:10:00.170228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d1ce8c-8143-405b-b095-d754313ce533 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:00.206544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d1ce8c-8143-405b-b095-d754313ce533 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:00.221830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876ea806-a31a-4437-b2b6-b025ef91f853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:00.235736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876ea806-a31a-4437-b2b6-b025ef91f853 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:00.235736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876ea806-a31a-4437-b2b6-b025ef91f853 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:00.235736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876ea806-a31a-4437-b2b6-b025ef91f853 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:00.238419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876ea806-a31a-4437-b2b6-b025ef91f853 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:00.243148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876ea806-a31a-4437-b2b6-b025ef91f853 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:00.254621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:00.263428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8297e2de-c616-4068-8d68-24398bba81af admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:00.273167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8297e2de-c616-4068-8d68-24398bba81af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:00.344119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1097/2191] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:10:00 2022] POST /identity/v3/projects => generated 365 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:10:00.349127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876ea806-a31a-4437-b2b6-b025ef91f853 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:00.368730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8297e2de-c616-4068-8d68-24398bba81af admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:00.502518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876ea806-a31a-4437-b2b6-b025ef91f853 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:00.534885 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e18975-b93b-4668-b3d9-f408a08592aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:00.543401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e18975-b93b-4668-b3d9-f408a08592aa admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:00.543401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e18975-b93b-4668-b3d9-f408a08592aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:00.543401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e18975-b93b-4668-b3d9-f408a08592aa admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:00.543401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e18975-b93b-4668-b3d9-f408a08592aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:00.554371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e18975-b93b-4668-b3d9-f408a08592aa admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:00.554990 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fcde5f-3553-4f83-a50a-51752f31a5d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:00.556362 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fcde5f-3553-4f83-a50a-51752f31a5d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:00.557080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fcde5f-3553-4f83-a50a-51752f31a5d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:00.558168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fcde5f-3553-4f83-a50a-51752f31a5d6 admin admin] PATH_INFO: `/v3/projects/55e30b5482384ba7902f98657f155a68` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:00.562854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fcde5f-3553-4f83-a50a-51752f31a5d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=55e30b5482384ba7902f98657f155a68)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:00.566856 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fcde5f-3553-4f83-a50a-51752f31a5d6 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:00.582735 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1098/2192] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:10:00 2022] POST /identity/v3/users => generated 283 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:10:00.586553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e18975-b93b-4668-b3d9-f408a08592aa admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:00.604904 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e18975-b93b-4668-b3d9-f408a08592aa admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:00.611317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eeb22db-4e7b-46e0-9941-c3883f7aaca6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:00.612138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eeb22db-4e7b-46e0-9941-c3883f7aaca6 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:00.612407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eeb22db-4e7b-46e0-9941-c3883f7aaca6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:00.612921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:00.613150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eeb22db-4e7b-46e0-9941-c3883f7aaca6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:00.614414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eeb22db-4e7b-46e0-9941-c3883f7aaca6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:00.616984 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eeb22db-4e7b-46e0-9941-c3883f7aaca6 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:00.631902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1099/2193] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:00 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:10:00.638187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eeb22db-4e7b-46e0-9941-c3883f7aaca6 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:00.650384 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eeb22db-4e7b-46e0-9941-c3883f7aaca6 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:00.668356 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b0ce93-14ec-4d9f-aae4-68e9abe2f4ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:00.669620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b0ce93-14ec-4d9f-aae4-68e9abe2f4ba admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:00.670001 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b0ce93-14ec-4d9f-aae4-68e9abe2f4ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:00.670378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b0ce93-14ec-4d9f-aae4-68e9abe2f4ba admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:00.671811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b0ce93-14ec-4d9f-aae4-68e9abe2f4ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:00.673632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-35fcde5f-3553-4f83-a50a-51752f31a5d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55e30b5482384ba7902f98657f155a68', 'request_id': 'req-35fcde5f-3553-4f83-a50a-51752f31a5d6'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:00.677219 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b0ce93-14ec-4d9f-aae4-68e9abe2f4ba admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:00.682468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-35fcde5f-3553-4f83-a50a-51752f31a5d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55e30b5482384ba7902f98657f155a68', 'request_id': 'req-35fcde5f-3553-4f83-a50a-51752f31a5d6'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:00.695103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-35fcde5f-3553-4f83-a50a-51752f31a5d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55e30b5482384ba7902f98657f155a68', 'request_id': 'req-35fcde5f-3553-4f83-a50a-51752f31a5d6'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:00.695408 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1100/2194] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:00 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:00.699732 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b0ce93-14ec-4d9f-aae4-68e9abe2f4ba admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:00.715777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-35fcde5f-3553-4f83-a50a-51752f31a5d6 admin admin] The token cache is being invalidate because project 55e30b5482384ba7902f98657f155a68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:00.716160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-35fcde5f-3553-4f83-a50a-51752f31a5d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:00.716962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-35fcde5f-3553-4f83-a50a-51752f31a5d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:00.718910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b0ce93-14ec-4d9f-aae4-68e9abe2f4ba admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:00.719485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1095/2195] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:00 2022] DELETE /identity/v3/projects/55e30b5482384ba7902f98657f155a68 => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:00.724642 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0b7291-0f24-4c6d-a7e9-19219cf3ffb7 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:00.724889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0b7291-0f24-4c6d-a7e9-19219cf3ffb7 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:00.725165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0b7291-0f24-4c6d-a7e9-19219cf3ffb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:00.749011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-1d0b7291-0f24-4c6d-a7e9-19219cf3ffb7 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:00.750460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1d0b7291-0f24-4c6d-a7e9-19219cf3ffb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:00.819290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5ad312-5db8-41d5-acfd-f66bdd3c9773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:00.820620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5ad312-5db8-41d5-acfd-f66bdd3c9773 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:00.820911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5ad312-5db8-41d5-acfd-f66bdd3c9773 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:00.821332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5ad312-5db8-41d5-acfd-f66bdd3c9773 admin admin] PATH_INFO: `/v3/projects/98680cfa86d54edea438fa624fbbb5d6/users/f8406100f2a04ea2981805d692094f17/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:00.840784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1096/2196] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:00.843143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2f7eeb-82c2-4c7a-b743-72d128184ee7 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:00.843479 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2f7eeb-82c2-4c7a-b743-72d128184ee7 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:00.844000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2f7eeb-82c2-4c7a-b743-72d128184ee7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:00.850566 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5ad312-5db8-41d5-acfd-f66bdd3c9773 admin admin] RBAC: Authorizing `identity:create_grant(project_id=98680cfa86d54edea438fa624fbbb5d6, user_id=f8406100f2a04ea2981805d692094f17, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:00.860968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5ad312-5db8-41d5-acfd-f66bdd3c9773 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:00.874823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-cf2f7eeb-82c2-4c7a-b743-72d128184ee7 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:00.876697 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-cf2f7eeb-82c2-4c7a-b743-72d128184ee7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:00.879877 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1101/2197] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:00 2022] PUT /identity/v3/projects/98680cfa86d54edea438fa624fbbb5d6/users/f8406100f2a04ea2981805d692094f17/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:00.890604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f610c1d4-770f-4579-9a3a-cc924716a86d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:00.891084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f610c1d4-770f-4579-9a3a-cc924716a86d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:00.891418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f610c1d4-770f-4579-9a3a-cc924716a86d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:00.936390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f610c1d4-770f-4579-9a3a-cc924716a86d None None] MFA Rules not processed for user `f8406100f2a04ea2981805d692094f17`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:00.938074 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f610c1d4-770f-4579-9a3a-cc924716a86d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:00.988259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:01.005149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1097/2198] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:01.011979 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2f7eeb-82c2-4c7a-b743-72d128184ee7 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.033427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2f7eeb-82c2-4c7a-b743-72d128184ee7 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:01.041806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b799063-0b49-45a9-a339-d9fd5a79e3cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:01.042569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b799063-0b49-45a9-a339-d9fd5a79e3cb admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.042906 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b799063-0b49-45a9-a339-d9fd5a79e3cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.043239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b799063-0b49-45a9-a339-d9fd5a79e3cb admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.044209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b799063-0b49-45a9-a339-d9fd5a79e3cb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:10:01.044756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b799063-0b49-45a9-a339-d9fd5a79e3cb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:01.047285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b799063-0b49-45a9-a339-d9fd5a79e3cb admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:01.063409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1098/2199] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:10:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:10:01.067903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b799063-0b49-45a9-a339-d9fd5a79e3cb admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.086024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1102/2200] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:00 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:01.086325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b799063-0b49-45a9-a339-d9fd5a79e3cb admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:01.088074 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f610c1d4-770f-4579-9a3a-cc924716a86d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.090354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f610c1d4-770f-4579-9a3a-cc924716a86d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:01.095253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e1840a-a0ec-4183-afca-5e06df687068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:01.096449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e1840a-a0ec-4183-afca-5e06df687068 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.097244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e1840a-a0ec-4183-afca-5e06df687068 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.098527 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e1840a-a0ec-4183-afca-5e06df687068 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.099016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e1840a-a0ec-4183-afca-5e06df687068 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:01.102747 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e1840a-a0ec-4183-afca-5e06df687068 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:01.118142 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f610c1d4-770f-4579-9a3a-cc924716a86d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:01.129163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771ce90a-ae4c-48e4-9bfc-82c829388646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:01.130687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771ce90a-ae4c-48e4-9bfc-82c829388646 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.131715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771ce90a-ae4c-48e4-9bfc-82c829388646 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.132457 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771ce90a-ae4c-48e4-9bfc-82c829388646 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.133777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771ce90a-ae4c-48e4-9bfc-82c829388646 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:01.138039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771ce90a-ae4c-48e4-9bfc-82c829388646 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:01.147289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1099/2201] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:10:01 2022] POST /identity/v3/projects => generated 373 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:10:01.148546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e1840a-a0ec-4183-afca-5e06df687068 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.173552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:01.203486 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1103/2202] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:10:01 2022] POST /identity/v3/projects => generated 365 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:10:01.210919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771ce90a-ae4c-48e4-9bfc-82c829388646 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.269437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e1840a-a0ec-4183-afca-5e06df687068 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:01.319613 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771ce90a-ae4c-48e4-9bfc-82c829388646 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:01.328897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3086bdd-1337-464a-a0da-0ba6dbe0d55d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:01.329560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3086bdd-1337-464a-a0da-0ba6dbe0d55d admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.329818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3086bdd-1337-464a-a0da-0ba6dbe0d55d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.330881 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3086bdd-1337-464a-a0da-0ba6dbe0d55d admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.330881 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3086bdd-1337-464a-a0da-0ba6dbe0d55d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:01.333197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3086bdd-1337-464a-a0da-0ba6dbe0d55d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:01.370931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1104/2203] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:10:01 2022] POST /identity/v3/users => generated 289 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:10:01.378790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106aa8e4-9aa3-43b5-b9de-1f33c449018f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:01.379325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3086bdd-1337-464a-a0da-0ba6dbe0d55d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.380373 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106aa8e4-9aa3-43b5-b9de-1f33c449018f admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.381228 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106aa8e4-9aa3-43b5-b9de-1f33c449018f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.381765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106aa8e4-9aa3-43b5-b9de-1f33c449018f admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.383010 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106aa8e4-9aa3-43b5-b9de-1f33c449018f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:01.387046 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106aa8e4-9aa3-43b5-b9de-1f33c449018f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:01.416438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3086bdd-1337-464a-a0da-0ba6dbe0d55d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:01.425408 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f7d611-4705-4a76-93fa-6be528e4ce4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:01.426449 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f7d611-4705-4a76-93fa-6be528e4ce4c admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.427108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1100/2204] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:10:01 2022] POST /identity/v3/users => generated 287 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:10:01.434946 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f7d611-4705-4a76-93fa-6be528e4ce4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.434946 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f7d611-4705-4a76-93fa-6be528e4ce4c admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.437379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106aa8e4-9aa3-43b5-b9de-1f33c449018f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.437533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f7d611-4705-4a76-93fa-6be528e4ce4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:01.443994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f7d611-4705-4a76-93fa-6be528e4ce4c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:01.471700 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1105/2205] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:01 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:01.478980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106aa8e4-9aa3-43b5-b9de-1f33c449018f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:01.479762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f7d611-4705-4a76-93fa-6be528e4ce4c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.489469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0223d4b-6d34-4513-91c8-ff2f38becaf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:01.490283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0223d4b-6d34-4513-91c8-ff2f38becaf3 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.490590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0223d4b-6d34-4513-91c8-ff2f38becaf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.490846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0223d4b-6d34-4513-91c8-ff2f38becaf3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.491575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0223d4b-6d34-4513-91c8-ff2f38becaf3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:01.496038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0223d4b-6d34-4513-91c8-ff2f38becaf3 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:01.501632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f7d611-4705-4a76-93fa-6be528e4ce4c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:01.511665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925585c6-f29d-4183-b13e-14bfca73626d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:01.513442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925585c6-f29d-4183-b13e-14bfca73626d admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.513652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925585c6-f29d-4183-b13e-14bfca73626d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.514031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925585c6-f29d-4183-b13e-14bfca73626d admin admin] PATH_INFO: `/v3/projects/8ef0b76d99794831aeeff13e82eeadb0/users/75ffc1ff291040bb903bb25ba0240e5a/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.521619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1101/2206] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:01 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:01.539518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0223d4b-6d34-4513-91c8-ff2f38becaf3 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.577530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0223d4b-6d34-4513-91c8-ff2f38becaf3 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:01.588811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c0ee8d-e171-4c46-a13c-d24ce39301ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:01.589068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925585c6-f29d-4183-b13e-14bfca73626d admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ef0b76d99794831aeeff13e82eeadb0, user_id=75ffc1ff291040bb903bb25ba0240e5a, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:01.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c0ee8d-e171-4c46-a13c-d24ce39301ab admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.591801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c0ee8d-e171-4c46-a13c-d24ce39301ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.592577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c0ee8d-e171-4c46-a13c-d24ce39301ab admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.594401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c0ee8d-e171-4c46-a13c-d24ce39301ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:01.595097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925585c6-f29d-4183-b13e-14bfca73626d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:01.598243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c0ee8d-e171-4c46-a13c-d24ce39301ab admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:01.632563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1102/2207] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:01 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:01.642090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c0ee8d-e171-4c46-a13c-d24ce39301ab admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.654549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c0ee8d-e171-4c46-a13c-d24ce39301ab admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:01.656690 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1106/2208] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:01 2022] PUT /identity/v3/projects/8ef0b76d99794831aeeff13e82eeadb0/users/75ffc1ff291040bb903bb25ba0240e5a/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:10:01.658616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925585c6-f29d-4183-b13e-14bfca73626d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.747643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767e3c3b-fab3-4d29-b99c-44b7743655c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:01.749625 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767e3c3b-fab3-4d29-b99c-44b7743655c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.749625 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767e3c3b-fab3-4d29-b99c-44b7743655c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.750529 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767e3c3b-fab3-4d29-b99c-44b7743655c0 admin admin] PATH_INFO: `/v3/projects/de83743a559b4a289aa40c8d16641ee9/users/8fcc34c42bb64200ac5f40d770d6784b/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.770873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767e3c3b-fab3-4d29-b99c-44b7743655c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de83743a559b4a289aa40c8d16641ee9, user_id=8fcc34c42bb64200ac5f40d770d6784b, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:01.774037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767e3c3b-fab3-4d29-b99c-44b7743655c0 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:01.791649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925585c6-f29d-4183-b13e-14bfca73626d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:01.795274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1103/2209] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:01 2022] PUT /identity/v3/projects/de83743a559b4a289aa40c8d16641ee9/users/8fcc34c42bb64200ac5f40d770d6784b/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:10:01.800029 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:01.802321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc95dc47-2145-41b4-994c-f3afbef3535f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.802735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc95dc47-2145-41b4-994c-f3afbef3535f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.803975 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc95dc47-2145-41b4-994c-f3afbef3535f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.841562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-dc95dc47-2145-41b4-994c-f3afbef3535f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:01.843392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-dc95dc47-2145-41b4-994c-f3afbef3535f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:01.850042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:01.893282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b414b0-60fb-4cda-ba2f-c2225b5112b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:01.894124 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b414b0-60fb-4cda-ba2f-c2225b5112b1 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.894480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b414b0-60fb-4cda-ba2f-c2225b5112b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.894852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b414b0-60fb-4cda-ba2f-c2225b5112b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.895703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b414b0-60fb-4cda-ba2f-c2225b5112b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:01.898264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b414b0-60fb-4cda-ba2f-c2225b5112b1 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:01.928985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1107/2210] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:01 2022] GET /identity/v3/roles => generated 1420 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:01.930150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b414b0-60fb-4cda-ba2f-c2225b5112b1 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.942849 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b414b0-60fb-4cda-ba2f-c2225b5112b1 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:01.946252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1104/2211] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:01.951741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35c9b12-e29c-4131-80b6-91ccf386b6cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:01.952704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35c9b12-e29c-4131-80b6-91ccf386b6cf admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.953096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35c9b12-e29c-4131-80b6-91ccf386b6cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.953477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35c9b12-e29c-4131-80b6-91ccf386b6cf admin admin] PATH_INFO: `/v3/projects/8ef0b76d99794831aeeff13e82eeadb0/users/75ffc1ff291040bb903bb25ba0240e5a/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:01.956325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35c9b12-e29c-4131-80b6-91ccf386b6cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ef0b76d99794831aeeff13e82eeadb0, user_id=75ffc1ff291040bb903bb25ba0240e5a, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:01.959767 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35c9b12-e29c-4131-80b6-91ccf386b6cf admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:01.960025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc95dc47-2145-41b4-994c-f3afbef3535f None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:01.961825 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-dc95dc47-2145-41b4-994c-f3afbef3535f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:01.974407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1108/2212] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:01 2022] PUT /identity/v3/projects/8ef0b76d99794831aeeff13e82eeadb0/users/75ffc1ff291040bb903bb25ba0240e5a/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:01.990192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a4e6b5-fbe6-4f7a-bbd1-3fdd833c8db3 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:01.990192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a4e6b5-fbe6-4f7a-bbd1-3fdd833c8db3 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:01.990192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a4e6b5-fbe6-4f7a-bbd1-3fdd833c8db3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:02.030409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b9a4e6b5-fbe6-4f7a-bbd1-3fdd833c8db3 None None] MFA Rules not processed for user `75ffc1ff291040bb903bb25ba0240e5a`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:02.032108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b9a4e6b5-fbe6-4f7a-bbd1-3fdd833c8db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:02.114573 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc95dc47-2145-41b4-994c-f3afbef3535f None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:02.185687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1109/2213] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:01 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:02.204741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a4e6b5-fbe6-4f7a-bbd1-3fdd833c8db3 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:02.207191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b9a4e6b5-fbe6-4f7a-bbd1-3fdd833c8db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:02.245893 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7535b8b3-d4df-4d89-b93f-bab53171ddd4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:02.246768 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7535b8b3-d4df-4d89-b93f-bab53171ddd4 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:02.247115 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7535b8b3-d4df-4d89-b93f-bab53171ddd4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:02.247539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7535b8b3-d4df-4d89-b93f-bab53171ddd4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:02.340437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7535b8b3-d4df-4d89-b93f-bab53171ddd4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:02.343324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7535b8b3-d4df-4d89-b93f-bab53171ddd4 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:02.369921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1105/2214] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:10:02.409268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a4e6b5-fbe6-4f7a-bbd1-3fdd833c8db3 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:02.413432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c535bf-6d58-47ff-8db7-8e8180578c6b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:02.414447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c535bf-6d58-47ff-8db7-8e8180578c6b service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:02.414623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c535bf-6d58-47ff-8db7-8e8180578c6b service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:02.415039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c535bf-6d58-47ff-8db7-8e8180578c6b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:02.444139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c535bf-6d58-47ff-8db7-8e8180578c6b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:02.445842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c535bf-6d58-47ff-8db7-8e8180578c6b service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:02.471776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1110/2215] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:02 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:02.666999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7535b8b3-d4df-4d89-b93f-bab53171ddd4 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:02.670142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7535b8b3-d4df-4d89-b93f-bab53171ddd4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:02.695968 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7535b8b3-d4df-4d89-b93f-bab53171ddd4 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:02.704188 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af55b862-c1af-45e6-bf89-eb2b662e8989 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:02.705375 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af55b862-c1af-45e6-bf89-eb2b662e8989 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:02.705850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af55b862-c1af-45e6-bf89-eb2b662e8989 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:02.706322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af55b862-c1af-45e6-bf89-eb2b662e8989 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:02.739916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:02.912081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af55b862-c1af-45e6-bf89-eb2b662e8989 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:02.917205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af55b862-c1af-45e6-bf89-eb2b662e8989 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:02.934773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1106/2216] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:02 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 269 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:03.775947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084519e1-2751-4866-ba03-7b9cd4bb6004 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:03.776715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084519e1-2751-4866-ba03-7b9cd4bb6004 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:03.776874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084519e1-2751-4866-ba03-7b9cd4bb6004 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:03.911655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-084519e1-2751-4866-ba03-7b9cd4bb6004 None None] MFA Rules not processed for user `8fcc34c42bb64200ac5f40d770d6784b`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:03.913087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-084519e1-2751-4866-ba03-7b9cd4bb6004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:03.926367 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:04.028826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1111/2217] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:03 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:04.046754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af55b862-c1af-45e6-bf89-eb2b662e8989 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:04.078185 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af55b862-c1af-45e6-bf89-eb2b662e8989 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:04.102408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2570e064-e3a4-493b-a1f5-ef5a2c616db0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:04.103485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2570e064-e3a4-493b-a1f5-ef5a2c616db0 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:04.103989 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2570e064-e3a4-493b-a1f5-ef5a2c616db0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:04.104433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2570e064-e3a4-493b-a1f5-ef5a2c616db0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:04.124491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2570e064-e3a4-493b-a1f5-ef5a2c616db0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:04.128737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2570e064-e3a4-493b-a1f5-ef5a2c616db0 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:04.146352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1107/2218] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:04 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:05.182341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084519e1-2751-4866-ba03-7b9cd4bb6004 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:05.184507 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-084519e1-2751-4866-ba03-7b9cd4bb6004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:05.215893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084519e1-2751-4866-ba03-7b9cd4bb6004 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:05.221459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e04971-8ed1-4f3c-bf28-f0d610d00be4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:05.222706 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e04971-8ed1-4f3c-bf28-f0d610d00be4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:05.222980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e04971-8ed1-4f3c-bf28-f0d610d00be4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:05.223500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e04971-8ed1-4f3c-bf28-f0d610d00be4 admin admin] PATH_INFO: `/v3/users/c4a002047e3e42338e4f57ba49e877fb` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:05.225335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e04971-8ed1-4f3c-bf28-f0d610d00be4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4a002047e3e42338e4f57ba49e877fb)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:05.227491 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e04971-8ed1-4f3c-bf28-f0d610d00be4 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:05.288959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e6e04971-8ed1-4f3c-bf28-f0d610d00be4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4a002047e3e42338e4f57ba49e877fb'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:05.289770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e6e04971-8ed1-4f3c-bf28-f0d610d00be4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4a002047e3e42338e4f57ba49e877fb'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:05.300006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e6e04971-8ed1-4f3c-bf28-f0d610d00be4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4a002047e3e42338e4f57ba49e877fb'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:05.323788 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e6e04971-8ed1-4f3c-bf28-f0d610d00be4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4a002047e3e42338e4f57ba49e877fb'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:05.337738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e6e04971-8ed1-4f3c-bf28-f0d610d00be4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4a002047e3e42338e4f57ba49e877fb'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:05.349590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1112/2219] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:10:05 2022] DELETE /identity/v3/users/c4a002047e3e42338e4f57ba49e877fb => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:05.584367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2570e064-e3a4-493b-a1f5-ef5a2c616db0 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:05.599977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2570e064-e3a4-493b-a1f5-ef5a2c616db0 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:05.678253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26474c10-b0fe-45d8-8e6d-cec2a2e29891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:05.678917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26474c10-b0fe-45d8-8e6d-cec2a2e29891 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:05.679200 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26474c10-b0fe-45d8-8e6d-cec2a2e29891 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:05.679517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26474c10-b0fe-45d8-8e6d-cec2a2e29891 admin admin] PATH_INFO: `/v3/projects/66c7f61afd0d4650bd530eff18a2318b` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:05.680773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26474c10-b0fe-45d8-8e6d-cec2a2e29891 admin admin] RBAC: Authorizing `identity:delete_project(project_id=66c7f61afd0d4650bd530eff18a2318b)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:05.682759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26474c10-b0fe-45d8-8e6d-cec2a2e29891 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:05.746211 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-26474c10-b0fe-45d8-8e6d-cec2a2e29891 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66c7f61afd0d4650bd530eff18a2318b', 'request_id': 'req-26474c10-b0fe-45d8-8e6d-cec2a2e29891'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:05.756171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-26474c10-b0fe-45d8-8e6d-cec2a2e29891 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66c7f61afd0d4650bd530eff18a2318b', 'request_id': 'req-26474c10-b0fe-45d8-8e6d-cec2a2e29891'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:05.766562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-26474c10-b0fe-45d8-8e6d-cec2a2e29891 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66c7f61afd0d4650bd530eff18a2318b', 'request_id': 'req-26474c10-b0fe-45d8-8e6d-cec2a2e29891'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:05.789955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-26474c10-b0fe-45d8-8e6d-cec2a2e29891 admin admin] The token cache is being invalidate because project 66c7f61afd0d4650bd530eff18a2318b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:05.790458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-26474c10-b0fe-45d8-8e6d-cec2a2e29891 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:05.791627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-26474c10-b0fe-45d8-8e6d-cec2a2e29891 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:05.794331 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1108/2220] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:05 2022] DELETE /identity/v3/projects/66c7f61afd0d4650bd530eff18a2318b => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:05.802949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a26e682-5d52-430a-9b22-ae0f5a919412 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:05.803449 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a26e682-5d52-430a-9b22-ae0f5a919412 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:05.803910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a26e682-5d52-430a-9b22-ae0f5a919412 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:05.836545 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-7a26e682-5d52-430a-9b22-ae0f5a919412 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:05.837789 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7a26e682-5d52-430a-9b22-ae0f5a919412 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:05.916253 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1113/2221] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:05.925148 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9c1090-9e62-4e6c-a25c-d5919ab3e4f5 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:05.925425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9c1090-9e62-4e6c-a25c-d5919ab3e4f5 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:05.925828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9c1090-9e62-4e6c-a25c-d5919ab3e4f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:05.946860 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-2b9c1090-9e62-4e6c-a25c-d5919ab3e4f5 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:05.948142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2b9c1090-9e62-4e6c-a25c-d5919ab3e4f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:05.956314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1109/2222] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:05.960795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a26e682-5d52-430a-9b22-ae0f5a919412 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:05.974043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a26e682-5d52-430a-9b22-ae0f5a919412 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:05.980252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b751822c-812b-445a-b3e2-39428336d57f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:05.981032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b751822c-812b-445a-b3e2-39428336d57f admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:05.981432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b751822c-812b-445a-b3e2-39428336d57f admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:05.981869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b751822c-812b-445a-b3e2-39428336d57f admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:05.982816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b751822c-812b-445a-b3e2-39428336d57f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:10:05.984199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b751822c-812b-445a-b3e2-39428336d57f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:05.986245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b751822c-812b-445a-b3e2-39428336d57f admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:06.011797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1114/2223] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:10:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:10:06.023145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9c1090-9e62-4e6c-a25c-d5919ab3e4f5 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:06.029273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9c1090-9e62-4e6c-a25c-d5919ab3e4f5 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:06.038490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ce4b08-cc68-491a-bac1-a1ae7e2847d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:06.039541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ce4b08-cc68-491a-bac1-a1ae7e2847d7 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:06.040005 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ce4b08-cc68-491a-bac1-a1ae7e2847d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:06.040455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ce4b08-cc68-491a-bac1-a1ae7e2847d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:06.041620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ce4b08-cc68-491a-bac1-a1ae7e2847d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:06.045770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ce4b08-cc68-491a-bac1-a1ae7e2847d7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:06.095785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1110/2224] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:10:06 2022] POST /identity/v3/projects => generated 391 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:10:06.103861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b751822c-812b-445a-b3e2-39428336d57f admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:06.123325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b751822c-812b-445a-b3e2-39428336d57f admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:06.130171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:06.245787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390b5ca1-62bb-4606-8a3e-26df6fa2a527 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:06.246872 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390b5ca1-62bb-4606-8a3e-26df6fa2a527 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:06.247373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390b5ca1-62bb-4606-8a3e-26df6fa2a527 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:06.248060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390b5ca1-62bb-4606-8a3e-26df6fa2a527 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:06.249175 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390b5ca1-62bb-4606-8a3e-26df6fa2a527 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:06.252962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390b5ca1-62bb-4606-8a3e-26df6fa2a527 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:06.295812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1115/2225] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:10:06 2022] POST /identity/v3/users => generated 296 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:10:06.297771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ce4b08-cc68-491a-bac1-a1ae7e2847d7 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:06.305769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:06.315684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ce4b08-cc68-491a-bac1-a1ae7e2847d7 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:06.324804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798e5d94-8324-4fcc-a806-7f222c4eaad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:06.325836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798e5d94-8324-4fcc-a806-7f222c4eaad6 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:06.327503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798e5d94-8324-4fcc-a806-7f222c4eaad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:06.327503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798e5d94-8324-4fcc-a806-7f222c4eaad6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:06.328744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798e5d94-8324-4fcc-a806-7f222c4eaad6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:06.331719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798e5d94-8324-4fcc-a806-7f222c4eaad6 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:06.349577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1111/2226] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:06 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:06.353689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390b5ca1-62bb-4606-8a3e-26df6fa2a527 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:06.365420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390b5ca1-62bb-4606-8a3e-26df6fa2a527 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:06.372069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281c0864-a8f9-4bce-8c5f-4d97401db753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:06.372711 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281c0864-a8f9-4bce-8c5f-4d97401db753 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:06.372991 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281c0864-a8f9-4bce-8c5f-4d97401db753 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:06.373272 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281c0864-a8f9-4bce-8c5f-4d97401db753 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:06.374098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281c0864-a8f9-4bce-8c5f-4d97401db753 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:06.375979 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281c0864-a8f9-4bce-8c5f-4d97401db753 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:06.394373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1116/2227] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:06 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:06.403807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798e5d94-8324-4fcc-a806-7f222c4eaad6 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:06.415443 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798e5d94-8324-4fcc-a806-7f222c4eaad6 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:06.420864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50754af2-8306-4473-900d-c32acdff8777 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:06.421640 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50754af2-8306-4473-900d-c32acdff8777 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:06.422108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50754af2-8306-4473-900d-c32acdff8777 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:06.422481 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50754af2-8306-4473-900d-c32acdff8777 admin admin] PATH_INFO: `/v3/projects/3d6a76e846cb48bb91eab4da1c842f2b/users/52e5a4deed99442e8eed62f334aa670d/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:06.446417 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50754af2-8306-4473-900d-c32acdff8777 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d6a76e846cb48bb91eab4da1c842f2b, user_id=52e5a4deed99442e8eed62f334aa670d, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:06.449473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50754af2-8306-4473-900d-c32acdff8777 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:06.466782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1112/2228] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:06 2022] PUT /identity/v3/projects/3d6a76e846cb48bb91eab4da1c842f2b/users/52e5a4deed99442e8eed62f334aa670d/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:10:06.474038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc01bf7-0180-424e-87b5-135061a9283e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:06.474520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc01bf7-0180-424e-87b5-135061a9283e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:06.475036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc01bf7-0180-424e-87b5-135061a9283e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:06.514454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-5bc01bf7-0180-424e-87b5-135061a9283e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:06.516528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5bc01bf7-0180-424e-87b5-135061a9283e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:06.628377 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1117/2229] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:06.647410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50754af2-8306-4473-900d-c32acdff8777 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:06.650949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-50754af2-8306-4473-900d-c32acdff8777 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:06.777302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50754af2-8306-4473-900d-c32acdff8777 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:06.784060 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1f0e4c-062d-4c99-9d0a-ec0214e677e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:06.784877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1f0e4c-062d-4c99-9d0a-ec0214e677e0 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:06.785198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1f0e4c-062d-4c99-9d0a-ec0214e677e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:06.785606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1f0e4c-062d-4c99-9d0a-ec0214e677e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:06.799674 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1f0e4c-062d-4c99-9d0a-ec0214e677e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:06.801221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1f0e4c-062d-4c99-9d0a-ec0214e677e0 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:06.813853 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1113/2230] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:07.151656 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0572d162-d937-47c1-ab68-2ff3e300dbec None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:07.152112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0572d162-d937-47c1-ab68-2ff3e300dbec None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:07.152672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0572d162-d937-47c1-ab68-2ff3e300dbec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:07.181021 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-0572d162-d937-47c1-ab68-2ff3e300dbec None None] MFA Rules not processed for user `8fcc34c42bb64200ac5f40d770d6784b`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:07.183088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0572d162-d937-47c1-ab68-2ff3e300dbec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:07.306820 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1118/2231] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:07 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:13.015222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1f0e4c-062d-4c99-9d0a-ec0214e677e0 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:13.017743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5e1f0e4c-062d-4c99-9d0a-ec0214e677e0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:13.051515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1f0e4c-062d-4c99-9d0a-ec0214e677e0 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:13.060309 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1801262-2fe1-441d-bb70-54a1614eda2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:13.061948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1801262-2fe1-441d-bb70-54a1614eda2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:13.062520 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1801262-2fe1-441d-bb70-54a1614eda2f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:13.063616 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1801262-2fe1-441d-bb70-54a1614eda2f admin admin] PATH_INFO: `/v3/users/bbb4d22a094c45869374fce82c52d3b6` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:13.065381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1801262-2fe1-441d-bb70-54a1614eda2f admin admin] RBAC: Authorizing `identity:delete_user(user_id=bbb4d22a094c45869374fce82c52d3b6)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:13.068590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0572d162-d937-47c1-ab68-2ff3e300dbec None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:13.071130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0572d162-d937-47c1-ab68-2ff3e300dbec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:13.087702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1801262-2fe1-441d-bb70-54a1614eda2f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:13.101373 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0572d162-d937-47c1-ab68-2ff3e300dbec None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:13.120164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c676203-28d3-4c74-9a0f-1af8440ad211 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:13.123668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c676203-28d3-4c74-9a0f-1af8440ad211 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:13.127116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c676203-28d3-4c74-9a0f-1af8440ad211 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:13.128787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c676203-28d3-4c74-9a0f-1af8440ad211 admin admin] PATH_INFO: `/v3/users/f8406100f2a04ea2981805d692094f17` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:13.130996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c676203-28d3-4c74-9a0f-1af8440ad211 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f8406100f2a04ea2981805d692094f17)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:13.133290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c676203-28d3-4c74-9a0f-1af8440ad211 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:13.263219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a1801262-2fe1-441d-bb70-54a1614eda2f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bbb4d22a094c45869374fce82c52d3b6'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:13.300793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a1801262-2fe1-441d-bb70-54a1614eda2f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bbb4d22a094c45869374fce82c52d3b6'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:13.302053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a1801262-2fe1-441d-bb70-54a1614eda2f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bbb4d22a094c45869374fce82c52d3b6'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:13.304379 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:13.339496 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a1801262-2fe1-441d-bb70-54a1614eda2f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bbb4d22a094c45869374fce82c52d3b6'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:13.340508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a1801262-2fe1-441d-bb70-54a1614eda2f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bbb4d22a094c45869374fce82c52d3b6'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:13.376457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1114/2232] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:10:13 2022] DELETE /identity/v3/users/bbb4d22a094c45869374fce82c52d3b6 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:13.378495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ec501-8818-40ae-bf2f-9b6ea6db2259 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:13.379030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ec501-8818-40ae-bf2f-9b6ea6db2259 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:13.379621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ec501-8818-40ae-bf2f-9b6ea6db2259 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:13.411403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6c676203-28d3-4c74-9a0f-1af8440ad211 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f8406100f2a04ea2981805d692094f17'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:13.413407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6c676203-28d3-4c74-9a0f-1af8440ad211 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f8406100f2a04ea2981805d692094f17'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:13.439927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6c676203-28d3-4c74-9a0f-1af8440ad211 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f8406100f2a04ea2981805d692094f17'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:13.473671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-919ec501-8818-40ae-bf2f-9b6ea6db2259 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:13.477055 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-919ec501-8818-40ae-bf2f-9b6ea6db2259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:13.506349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6c676203-28d3-4c74-9a0f-1af8440ad211 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f8406100f2a04ea2981805d692094f17'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:13.604919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6c676203-28d3-4c74-9a0f-1af8440ad211 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f8406100f2a04ea2981805d692094f17'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:13.636560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1119/2233] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:10:13 2022] DELETE /identity/v3/users/f8406100f2a04ea2981805d692094f17 => generated 0 bytes in 569 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:13.643087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d408c00-6675-4fd1-98f3-64efd9eeecc8 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:13.643671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d408c00-6675-4fd1-98f3-64efd9eeecc8 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:13.644388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d408c00-6675-4fd1-98f3-64efd9eeecc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:13.662561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:13.713993 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-7d408c00-6675-4fd1-98f3-64efd9eeecc8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:13.715330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7d408c00-6675-4fd1-98f3-64efd9eeecc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:13.780333 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1115/2234] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 403 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:13.812103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ec501-8818-40ae-bf2f-9b6ea6db2259 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:13.813653 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-919ec501-8818-40ae-bf2f-9b6ea6db2259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:14.016637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1120/2235] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 374 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:14.065248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ec501-8818-40ae-bf2f-9b6ea6db2259 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:14.152480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d408c00-6675-4fd1-98f3-64efd9eeecc8 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:14.246076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309b2be5-a020-4d81-abe5-ac94efcd1801 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:14.247602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b2be5-a020-4d81-abe5-ac94efcd1801 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:14.248251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b2be5-a020-4d81-abe5-ac94efcd1801 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:14.250549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b2be5-a020-4d81-abe5-ac94efcd1801 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:14.281269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d408c00-6675-4fd1-98f3-64efd9eeecc8 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:14.288909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f418f9ba-4832-4388-ab2c-daa7d0d47e81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:14.289953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f418f9ba-4832-4388-ab2c-daa7d0d47e81 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:14.290234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f418f9ba-4832-4388-ab2c-daa7d0d47e81 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:14.290507 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f418f9ba-4832-4388-ab2c-daa7d0d47e81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:14.311419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f418f9ba-4832-4388-ab2c-daa7d0d47e81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:14.312933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f418f9ba-4832-4388-ab2c-daa7d0d47e81 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:14.317683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309b2be5-a020-4d81-abe5-ac94efcd1801 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:14.339442 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309b2be5-a020-4d81-abe5-ac94efcd1801 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:14.359166 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1121/2236] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:14.396071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1116/2237] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 585 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:14.926207 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ea32d9-4f91-458b-ab82-35d45a9cd0fd None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:14.926669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ea32d9-4f91-458b-ab82-35d45a9cd0fd None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:14.926828 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ea32d9-4f91-458b-ab82-35d45a9cd0fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:14.994493 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-46ea32d9-4f91-458b-ab82-35d45a9cd0fd None None] MFA Rules not processed for user `52e5a4deed99442e8eed62f334aa670d`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:14.998715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-46ea32d9-4f91-458b-ab82-35d45a9cd0fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:15.116779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:15.235745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1122/2238] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:14 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 310 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:15.256935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d97faad-078e-4ccb-91e3-ecb9a86e4d33 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:15.257769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d97faad-078e-4ccb-91e3-ecb9a86e4d33 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:15.257963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d97faad-078e-4ccb-91e3-ecb9a86e4d33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:15.360872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-8d97faad-078e-4ccb-91e3-ecb9a86e4d33 None None] MFA Rules not processed for user `52e5a4deed99442e8eed62f334aa670d`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:15.362352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8d97faad-078e-4ccb-91e3-ecb9a86e4d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:15.377560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1117/2239] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:15 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:15.407224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ea32d9-4f91-458b-ab82-35d45a9cd0fd None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:15.409842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-46ea32d9-4f91-458b-ab82-35d45a9cd0fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:15.624838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d97faad-078e-4ccb-91e3-ecb9a86e4d33 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:15.627476 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8d97faad-078e-4ccb-91e3-ecb9a86e4d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:15.656308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ea32d9-4f91-458b-ab82-35d45a9cd0fd None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:15.671226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7fef3c-be1f-4224-b155-9e68fcb9000d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:15.672873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7fef3c-be1f-4224-b155-9e68fcb9000d service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:15.673696 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7fef3c-be1f-4224-b155-9e68fcb9000d service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:15.674341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7fef3c-be1f-4224-b155-9e68fcb9000d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:15.687005 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d97faad-078e-4ccb-91e3-ecb9a86e4d33 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:15.710917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:15.712102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:15.712598 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:15.713046 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] PATH_INFO: `/v3/projects/a42ab953bdf2412a97e41e971d859971` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:15.716723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a42ab953bdf2412a97e41e971d859971)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:15.718452 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7fef3c-be1f-4224-b155-9e68fcb9000d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:15.721426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7fef3c-be1f-4224-b155-9e68fcb9000d service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:15.726074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:15.771173 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1123/2240] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:15 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:15.891127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7fef3c-be1f-4224-b155-9e68fcb9000d service nova] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:15.893945 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-aa7fef3c-be1f-4224-b155-9e68fcb9000d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:15.950067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a42ab953bdf2412a97e41e971d859971', 'request_id': 'req-2d6a6137-a624-4c3f-9dc2-d551075b40e1'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:15.956258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7fef3c-be1f-4224-b155-9e68fcb9000d service nova] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:15.977409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a42ab953bdf2412a97e41e971d859971', 'request_id': 'req-2d6a6137-a624-4c3f-9dc2-d551075b40e1'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.007523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a42ab953bdf2412a97e41e971d859971', 'request_id': 'req-2d6a6137-a624-4c3f-9dc2-d551075b40e1'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.059130 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] The token cache is being invalidate because project a42ab953bdf2412a97e41e971d859971 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:16.059726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.063866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.069811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1118/2241] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:15 2022] DELETE /identity/v3/projects/a42ab953bdf2412a97e41e971d859971 => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:10:16.075249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:16.093230 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:16.172196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:16.195010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:16.196076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:16.196550 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:16.197007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] PATH_INFO: `/v3/projects/98680cfa86d54edea438fa624fbbb5d6` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:16.199387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] RBAC: Authorizing `identity:delete_project(project_id=98680cfa86d54edea438fa624fbbb5d6)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:16.205055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:16.354525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98680cfa86d54edea438fa624fbbb5d6', 'request_id': 'req-82cc2fc1-3c25-42c6-b610-739bc74bd73a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.366911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a6137-a624-4c3f-9dc2-d551075b40e1 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:16.367967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98680cfa86d54edea438fa624fbbb5d6', 'request_id': 'req-82cc2fc1-3c25-42c6-b610-739bc74bd73a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.392945 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98680cfa86d54edea438fa624fbbb5d6', 'request_id': 'req-82cc2fc1-3c25-42c6-b610-739bc74bd73a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.399593 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:16.484194 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] The token cache is being invalidate because project 98680cfa86d54edea438fa624fbbb5d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:16.485095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.487024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.495432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1124/2242] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:15 2022] DELETE /identity/v3/projects/98680cfa86d54edea438fa624fbbb5d6 => generated 0 bytes in 606 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:10:16.500708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:16.504792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:16.594321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:16.596857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:16.597618 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:16.599181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] PATH_INFO: `/v3/users/dbf0ad05e9884a2b9edb7e5fc40fd536` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:16.604020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dbf0ad05e9884a2b9edb7e5fc40fd536)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:16.610675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:16.655991 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cc2fc1-3c25-42c6-b610-739bc74bd73a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:16.663484 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:16.664759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:16.665451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:16.665879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] PATH_INFO: `/v3/users/75ffc1ff291040bb903bb25ba0240e5a` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:16.668755 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75ffc1ff291040bb903bb25ba0240e5a)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:16.693088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:16.781413 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dbf0ad05e9884a2b9edb7e5fc40fd536'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.796490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dbf0ad05e9884a2b9edb7e5fc40fd536'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.798100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dbf0ad05e9884a2b9edb7e5fc40fd536'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.824295 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dbf0ad05e9884a2b9edb7e5fc40fd536'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.824558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75ffc1ff291040bb903bb25ba0240e5a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.825268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dbf0ad05e9884a2b9edb7e5fc40fd536'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.825930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75ffc1ff291040bb903bb25ba0240e5a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.842288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75ffc1ff291040bb903bb25ba0240e5a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.863326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75ffc1ff291040bb903bb25ba0240e5a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.868889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1119/2243] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:10:16 2022] DELETE /identity/v3/users/dbf0ad05e9884a2b9edb7e5fc40fd536 => generated 0 bytes in 794 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:16.890555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75ffc1ff291040bb903bb25ba0240e5a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:16.915084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1125/2244] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:10:16 2022] DELETE /identity/v3/users/75ffc1ff291040bb903bb25ba0240e5a => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:17.664798 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:17.669325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:17.720068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:17.723077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:17.842831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:17.877374 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fc4f4a-63b5-4ebc-bde2-f6edd636d4c5 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:17.893561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04a805b-b832-49eb-ab6e-9697dad9cca3 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:17.899898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5754391-2325-4d37-be26-e3c3356037ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:17.900599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5754391-2325-4d37-be26-e3c3356037ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:17.900850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5754391-2325-4d37-be26-e3c3356037ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:17.901128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5754391-2325-4d37-be26-e3c3356037ed admin admin] PATH_INFO: `/v3/projects/8ef0b76d99794831aeeff13e82eeadb0` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:17.902427 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5754391-2325-4d37-be26-e3c3356037ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ef0b76d99794831aeeff13e82eeadb0)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:17.906093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5754391-2325-4d37-be26-e3c3356037ed admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:17.921203 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:17.992650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68718fc-3d4f-4790-be11-6288792cf6d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:17.994256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68718fc-3d4f-4790-be11-6288792cf6d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:17.994841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68718fc-3d4f-4790-be11-6288792cf6d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:17.995202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68718fc-3d4f-4790-be11-6288792cf6d7 admin admin] PATH_INFO: `/v3/projects/32c56170c7ec421196cc95704d1f136f` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:17.998381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68718fc-3d4f-4790-be11-6288792cf6d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=32c56170c7ec421196cc95704d1f136f)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:18.002503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68718fc-3d4f-4790-be11-6288792cf6d7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:18.005510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f5754391-2325-4d37-be26-e3c3356037ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ef0b76d99794831aeeff13e82eeadb0', 'request_id': 'req-f5754391-2325-4d37-be26-e3c3356037ed'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:18.013262 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f5754391-2325-4d37-be26-e3c3356037ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ef0b76d99794831aeeff13e82eeadb0', 'request_id': 'req-f5754391-2325-4d37-be26-e3c3356037ed'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:18.033731 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f5754391-2325-4d37-be26-e3c3356037ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ef0b76d99794831aeeff13e82eeadb0', 'request_id': 'req-f5754391-2325-4d37-be26-e3c3356037ed'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:18.051534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f5754391-2325-4d37-be26-e3c3356037ed admin admin] The token cache is being invalidate because project 8ef0b76d99794831aeeff13e82eeadb0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:18.051842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f5754391-2325-4d37-be26-e3c3356037ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:18.053436 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f5754391-2325-4d37-be26-e3c3356037ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:18.059747 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1126/2245] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:17 2022] DELETE /identity/v3/projects/8ef0b76d99794831aeeff13e82eeadb0 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:18.062109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2032cedd-7db9-4e1e-9bc0-1901f3fe5291 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:18.062650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2032cedd-7db9-4e1e-9bc0-1901f3fe5291 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:18.062937 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2032cedd-7db9-4e1e-9bc0-1901f3fe5291 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:18.110015 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-2032cedd-7db9-4e1e-9bc0-1901f3fe5291 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:18.111274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2032cedd-7db9-4e1e-9bc0-1901f3fe5291 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:18.120108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-b68718fc-3d4f-4790-be11-6288792cf6d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32c56170c7ec421196cc95704d1f136f', 'request_id': 'req-b68718fc-3d4f-4790-be11-6288792cf6d7'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:18.132990 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-b68718fc-3d4f-4790-be11-6288792cf6d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32c56170c7ec421196cc95704d1f136f', 'request_id': 'req-b68718fc-3d4f-4790-be11-6288792cf6d7'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:18.141058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-b68718fc-3d4f-4790-be11-6288792cf6d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32c56170c7ec421196cc95704d1f136f', 'request_id': 'req-b68718fc-3d4f-4790-be11-6288792cf6d7'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:18.170425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-b68718fc-3d4f-4790-be11-6288792cf6d7 admin admin] The token cache is being invalidate because project 32c56170c7ec421196cc95704d1f136f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:18.170425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-b68718fc-3d4f-4790-be11-6288792cf6d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:18.171768 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-b68718fc-3d4f-4790-be11-6288792cf6d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:18.173971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1120/2246] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:17 2022] DELETE /identity/v3/projects/32c56170c7ec421196cc95704d1f136f => generated 0 bytes in 511 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:18.183242 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2284ed-d20a-4cda-9a55-ceffd4753a6d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:18.183933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2284ed-d20a-4cda-9a55-ceffd4753a6d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:18.184216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2284ed-d20a-4cda-9a55-ceffd4753a6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:18.212518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-0f2284ed-d20a-4cda-9a55-ceffd4753a6d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:18.213877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0f2284ed-d20a-4cda-9a55-ceffd4753a6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:18.330312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1127/2247] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:18.338044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635ba061-1e66-4a73-8601-d221cae7dc6d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:18.338485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635ba061-1e66-4a73-8601-d221cae7dc6d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:18.338971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635ba061-1e66-4a73-8601-d221cae7dc6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:18.376402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1121/2248] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:18.381712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad39c58-d7bc-4218-b4e3-c7f2326015e8 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:18.382350 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad39c58-d7bc-4218-b4e3-c7f2326015e8 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:18.382909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad39c58-d7bc-4218-b4e3-c7f2326015e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:18.398128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-635ba061-1e66-4a73-8601-d221cae7dc6d None None] MFA Rules not processed for user `52e5a4deed99442e8eed62f334aa670d`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:18.400277 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-635ba061-1e66-4a73-8601-d221cae7dc6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:18.450841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-0ad39c58-d7bc-4218-b4e3-c7f2326015e8 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:18.452574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0ad39c58-d7bc-4218-b4e3-c7f2326015e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:18.465213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1122/2249] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:18.465213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b74852e-f734-4f43-9e89-5bfa436ffccb None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:18.465213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b74852e-f734-4f43-9e89-5bfa436ffccb None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:18.465213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b74852e-f734-4f43-9e89-5bfa436ffccb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:18.488123 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-1b74852e-f734-4f43-9e89-5bfa436ffccb None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:18.491102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1b74852e-f734-4f43-9e89-5bfa436ffccb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:18.509683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1123/2250] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:18.514180 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b74852e-f734-4f43-9e89-5bfa436ffccb None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:18.531965 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b74852e-f734-4f43-9e89-5bfa436ffccb None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:18.540290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75878018-e4b0-4056-845b-7be5851ac7db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:18.540986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75878018-e4b0-4056-845b-7be5851ac7db admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:18.541303 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75878018-e4b0-4056-845b-7be5851ac7db admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:18.541603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75878018-e4b0-4056-845b-7be5851ac7db admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:18.542314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75878018-e4b0-4056-845b-7be5851ac7db admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:10:18.542616 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75878018-e4b0-4056-845b-7be5851ac7db admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:18.552333 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75878018-e4b0-4056-845b-7be5851ac7db admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:18.557844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:18.567582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1128/2251] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:18 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:18.567582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635ba061-1e66-4a73-8601-d221cae7dc6d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:18.581236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1124/2252] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:10:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:10:18.582975 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75878018-e4b0-4056-845b-7be5851ac7db admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:18.599177 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75878018-e4b0-4056-845b-7be5851ac7db admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:18.609004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31eeac4-8bda-4043-91f9-bfd3996c6be2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:18.609743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31eeac4-8bda-4043-91f9-bfd3996c6be2 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:18.610093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31eeac4-8bda-4043-91f9-bfd3996c6be2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:18.610437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31eeac4-8bda-4043-91f9-bfd3996c6be2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:18.612468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31eeac4-8bda-4043-91f9-bfd3996c6be2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:18.615465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31eeac4-8bda-4043-91f9-bfd3996c6be2 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:18.626824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635ba061-1e66-4a73-8601-d221cae7dc6d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:18.641407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a164bf28-9aa0-4cb3-a610-489837abc5f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:18.642819 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a164bf28-9aa0-4cb3-a610-489837abc5f9 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:18.643925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a164bf28-9aa0-4cb3-a610-489837abc5f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:18.645694 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a164bf28-9aa0-4cb3-a610-489837abc5f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:18.648128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a164bf28-9aa0-4cb3-a610-489837abc5f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:10:18.648994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a164bf28-9aa0-4cb3-a610-489837abc5f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:18.674507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1125/2253] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:10:18 2022] POST /identity/v3/projects => generated 387 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:10:18.679811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31eeac4-8bda-4043-91f9-bfd3996c6be2 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:18.682574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a164bf28-9aa0-4cb3-a610-489837abc5f9 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:18.738825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1129/2254] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:10:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:10:18.756741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a164bf28-9aa0-4cb3-a610-489837abc5f9 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:18.787837 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a164bf28-9aa0-4cb3-a610-489837abc5f9 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:18.850640 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31eeac4-8bda-4043-91f9-bfd3996c6be2 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:18.858113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0a47cb-b99e-4ab0-99a5-70f4a6edb15a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:18.859020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0a47cb-b99e-4ab0-99a5-70f4a6edb15a admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:18.859303 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0a47cb-b99e-4ab0-99a5-70f4a6edb15a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:18.859755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0a47cb-b99e-4ab0-99a5-70f4a6edb15a admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:18.860706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0a47cb-b99e-4ab0-99a5-70f4a6edb15a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:18.863000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0a47cb-b99e-4ab0-99a5-70f4a6edb15a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:18.922525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:18.924006 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1126/2255] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:10:18 2022] POST /identity/v3/users => generated 294 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:10:18.925997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0a47cb-b99e-4ab0-99a5-70f4a6edb15a admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:18.947993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0a47cb-b99e-4ab0-99a5-70f4a6edb15a admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:18.958186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe455694-8e5d-4b6d-b4d3-acc191835a2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:18.959373 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe455694-8e5d-4b6d-b4d3-acc191835a2e admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:18.959868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe455694-8e5d-4b6d-b4d3-acc191835a2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:18.960379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe455694-8e5d-4b6d-b4d3-acc191835a2e admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:18.961686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe455694-8e5d-4b6d-b4d3-acc191835a2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:18.966228 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe455694-8e5d-4b6d-b4d3-acc191835a2e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:18.984679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1127/2256] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:18 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:18.986310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e5448b-dae8-49ec-97bc-b3b85b1d5d98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:18.987430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e5448b-dae8-49ec-97bc-b3b85b1d5d98 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:18.988064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e5448b-dae8-49ec-97bc-b3b85b1d5d98 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:18.988421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e5448b-dae8-49ec-97bc-b3b85b1d5d98 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:18.989594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e5448b-dae8-49ec-97bc-b3b85b1d5d98 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:18.989797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe455694-8e5d-4b6d-b4d3-acc191835a2e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:18.993775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e5448b-dae8-49ec-97bc-b3b85b1d5d98 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:19.004681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe455694-8e5d-4b6d-b4d3-acc191835a2e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:19.010457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567ceab7-d159-4b78-8e4e-ef231e984da4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:19.011179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567ceab7-d159-4b78-8e4e-ef231e984da4 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:19.011553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567ceab7-d159-4b78-8e4e-ef231e984da4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:19.011923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567ceab7-d159-4b78-8e4e-ef231e984da4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:19.012827 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567ceab7-d159-4b78-8e4e-ef231e984da4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:19.015122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567ceab7-d159-4b78-8e4e-ef231e984da4 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:19.035771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1128/2257] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:18 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:19.041426 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567ceab7-d159-4b78-8e4e-ef231e984da4 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:19.051774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1130/2258] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:10:18 2022] POST /identity/v3/projects => generated 371 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:10:19.054092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567ceab7-d159-4b78-8e4e-ef231e984da4 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:19.057785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e5448b-dae8-49ec-97bc-b3b85b1d5d98 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:19.199787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427a8315-4847-47d6-80e3-190a10182aa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:19.201327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427a8315-4847-47d6-80e3-190a10182aa4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:19.202173 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427a8315-4847-47d6-80e3-190a10182aa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:19.202644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427a8315-4847-47d6-80e3-190a10182aa4 admin admin] PATH_INFO: `/v3/projects/6e84f8b62a0e484db45b816e65055e8f/users/d794705f3112451abe273ac7265ad053/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:19.215249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e5448b-dae8-49ec-97bc-b3b85b1d5d98 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:19.224697 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8155d1d-2a56-4f40-b376-d07b098f6886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:19.225981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8155d1d-2a56-4f40-b376-d07b098f6886 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:19.226695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8155d1d-2a56-4f40-b376-d07b098f6886 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:19.227241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8155d1d-2a56-4f40-b376-d07b098f6886 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:19.228602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8155d1d-2a56-4f40-b376-d07b098f6886 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:19.237210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8155d1d-2a56-4f40-b376-d07b098f6886 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:19.238729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427a8315-4847-47d6-80e3-190a10182aa4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e84f8b62a0e484db45b816e65055e8f, user_id=d794705f3112451abe273ac7265ad053, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:19.244503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427a8315-4847-47d6-80e3-190a10182aa4 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:19.275884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1129/2259] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:19 2022] PUT /identity/v3/projects/6e84f8b62a0e484db45b816e65055e8f/users/d794705f3112451abe273ac7265ad053/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:19.297923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1131/2260] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:10:19 2022] POST /identity/v3/users => generated 286 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:10:19.300509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65113b0-f06b-4b21-b752-774d2572dc49 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:19.301260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65113b0-f06b-4b21-b752-774d2572dc49 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:19.301955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65113b0-f06b-4b21-b752-774d2572dc49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:19.305214 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8155d1d-2a56-4f40-b376-d07b098f6886 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:19.330790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8155d1d-2a56-4f40-b376-d07b098f6886 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:19.362333 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-c65113b0-f06b-4b21-b752-774d2572dc49 None None] MFA Rules not processed for user `d794705f3112451abe273ac7265ad053`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:19.364730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c65113b0-f06b-4b21-b752-774d2572dc49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:19.390374 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:19.530179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1130/2261] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:19 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:19.536509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65113b0-f06b-4b21-b752-774d2572dc49 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:19.586408 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4268f369-718c-4f9b-9a0d-af2a3075c062 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:19.587955 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4268f369-718c-4f9b-9a0d-af2a3075c062 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:19.588465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4268f369-718c-4f9b-9a0d-af2a3075c062 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:19.588967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4268f369-718c-4f9b-9a0d-af2a3075c062 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:19.590392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4268f369-718c-4f9b-9a0d-af2a3075c062 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:19.596411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4268f369-718c-4f9b-9a0d-af2a3075c062 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:19.660363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1132/2262] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:19 2022] GET /identity/v3/roles => generated 1420 bytes in 352 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:19.663059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4268f369-718c-4f9b-9a0d-af2a3075c062 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:19.713291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4268f369-718c-4f9b-9a0d-af2a3075c062 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:19.739470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9056051b-3169-4521-9f53-535b08e3a32d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:19.745138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9056051b-3169-4521-9f53-535b08e3a32d admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:19.747857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9056051b-3169-4521-9f53-535b08e3a32d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:19.749662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9056051b-3169-4521-9f53-535b08e3a32d admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:19.755828 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9056051b-3169-4521-9f53-535b08e3a32d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:19.762331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9056051b-3169-4521-9f53-535b08e3a32d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:19.772076 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65113b0-f06b-4b21-b752-774d2572dc49 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:19.795468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1133/2263] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:19 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:19.802399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9056051b-3169-4521-9f53-535b08e3a32d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:19.802707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41199316-3c1d-4bbb-bdbe-ff1334114acb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:19.808922 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41199316-3c1d-4bbb-bdbe-ff1334114acb admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:19.809525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41199316-3c1d-4bbb-bdbe-ff1334114acb admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:19.810230 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41199316-3c1d-4bbb-bdbe-ff1334114acb admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:19.811670 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41199316-3c1d-4bbb-bdbe-ff1334114acb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:19.821283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41199316-3c1d-4bbb-bdbe-ff1334114acb admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:19.827347 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9056051b-3169-4521-9f53-535b08e3a32d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:19.844319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c102c-4982-4fb0-b3ef-2be9eb501779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:19.845156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c102c-4982-4fb0-b3ef-2be9eb501779 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:19.845526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c102c-4982-4fb0-b3ef-2be9eb501779 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:19.845888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c102c-4982-4fb0-b3ef-2be9eb501779 admin admin] PATH_INFO: `/v3/projects/8928680faee44bf4a612a289e3c79665/users/1578277a38e84d42b5c56718e895f251/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:19.869435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c102c-4982-4fb0-b3ef-2be9eb501779 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8928680faee44bf4a612a289e3c79665, user_id=1578277a38e84d42b5c56718e895f251, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:19.872441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c102c-4982-4fb0-b3ef-2be9eb501779 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:19.906475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1134/2264] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:19 2022] PUT /identity/v3/projects/8928680faee44bf4a612a289e3c79665/users/1578277a38e84d42b5c56718e895f251/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:19.934109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5a4fe2-1b7f-40d4-a559-ddaccfceb7ac None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:19.934366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5a4fe2-1b7f-40d4-a559-ddaccfceb7ac None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:19.934646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5a4fe2-1b7f-40d4-a559-ddaccfceb7ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:19.947972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1131/2265] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:10:19 2022] POST /identity/v3/projects => generated 385 bytes in 412 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:10:19.953967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41199316-3c1d-4bbb-bdbe-ff1334114acb admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:19.978774 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:20.002771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-cd5a4fe2-1b7f-40d4-a559-ddaccfceb7ac None None] MFA Rules not processed for user `1578277a38e84d42b5c56718e895f251`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:20.005086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-cd5a4fe2-1b7f-40d4-a559-ddaccfceb7ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:20.142276 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41199316-3c1d-4bbb-bdbe-ff1334114acb admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:20.159817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f7cfdd-0867-4ff6-8421-0b7b8a258902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:20.164935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f7cfdd-0867-4ff6-8421-0b7b8a258902 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:20.166222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f7cfdd-0867-4ff6-8421-0b7b8a258902 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:20.166834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f7cfdd-0867-4ff6-8421-0b7b8a258902 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:20.168470 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f7cfdd-0867-4ff6-8421-0b7b8a258902 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:20.175092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f7cfdd-0867-4ff6-8421-0b7b8a258902 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:20.204617 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1135/2266] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:19 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:20.228977 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5a4fe2-1b7f-40d4-a559-ddaccfceb7ac None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:20.230865 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-cd5a4fe2-1b7f-40d4-a559-ddaccfceb7ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:20.243000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1132/2267] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:10:19 2022] POST /identity/v3/users => generated 299 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:10:20.248733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f7cfdd-0867-4ff6-8421-0b7b8a258902 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:20.291675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f7cfdd-0867-4ff6-8421-0b7b8a258902 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:20.300166 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a17b38-0a05-4d8c-9629-75402941eb64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:20.301726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a17b38-0a05-4d8c-9629-75402941eb64 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:20.302738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a17b38-0a05-4d8c-9629-75402941eb64 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:20.303559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a17b38-0a05-4d8c-9629-75402941eb64 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:20.305239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a17b38-0a05-4d8c-9629-75402941eb64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:20.309405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a17b38-0a05-4d8c-9629-75402941eb64 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:20.335130 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1133/2268] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:20 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:20.343034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a17b38-0a05-4d8c-9629-75402941eb64 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:20.362515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a17b38-0a05-4d8c-9629-75402941eb64 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:20.371613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8be05f3-c818-4d39-9a3d-45496922854f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:20.372515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8be05f3-c818-4d39-9a3d-45496922854f admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:20.373635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8be05f3-c818-4d39-9a3d-45496922854f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:20.374151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8be05f3-c818-4d39-9a3d-45496922854f admin admin] PATH_INFO: `/v3/projects/5c02f855b54c41f1a0c859eed726d0ed/users/520331d6b72a408491fa1bf4eb9a0b13/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:20.412092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5a4fe2-1b7f-40d4-a559-ddaccfceb7ac None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:20.414528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8be05f3-c818-4d39-9a3d-45496922854f admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c02f855b54c41f1a0c859eed726d0ed, user_id=520331d6b72a408491fa1bf4eb9a0b13, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:20.419051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba02e858-0ee9-4304-8018-95ffb174fd0a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:20.420252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba02e858-0ee9-4304-8018-95ffb174fd0a service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:20.420710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba02e858-0ee9-4304-8018-95ffb174fd0a service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:20.421153 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba02e858-0ee9-4304-8018-95ffb174fd0a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:20.422308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8be05f3-c818-4d39-9a3d-45496922854f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:20.427637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:20.440811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba02e858-0ee9-4304-8018-95ffb174fd0a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:20.443788 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba02e858-0ee9-4304-8018-95ffb174fd0a service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:20.463573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1136/2270] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:20 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:20.464660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1134/2270] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:20 2022] PUT /identity/v3/projects/5c02f855b54c41f1a0c859eed726d0ed/users/520331d6b72a408491fa1bf4eb9a0b13/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:20.465135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba02e858-0ee9-4304-8018-95ffb174fd0a service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:20.492930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba02e858-0ee9-4304-8018-95ffb174fd0a service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:20.511124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8be05f3-c818-4d39-9a3d-45496922854f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:20.673217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f9d5b7-f7c4-48b2-8086-8369032d55c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:20.674367 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f9d5b7-f7c4-48b2-8086-8369032d55c9 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:20.675412 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f9d5b7-f7c4-48b2-8086-8369032d55c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:20.675684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f9d5b7-f7c4-48b2-8086-8369032d55c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:20.677127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f9d5b7-f7c4-48b2-8086-8369032d55c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:20.680095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f9d5b7-f7c4-48b2-8086-8369032d55c9 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:20.699699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8be05f3-c818-4d39-9a3d-45496922854f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:20.703242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1137/2271] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:20 2022] GET /identity/v3/roles => generated 1420 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:20.707960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee3a25a-e97e-4166-afac-ecab33aa21ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:20.709155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee3a25a-e97e-4166-afac-ecab33aa21ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:20.709792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f9d5b7-f7c4-48b2-8086-8369032d55c9 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:20.710062 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee3a25a-e97e-4166-afac-ecab33aa21ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:20.710641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee3a25a-e97e-4166-afac-ecab33aa21ac admin admin] PATH_INFO: `/v3/users/1578277a38e84d42b5c56718e895f251` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:20.712504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee3a25a-e97e-4166-afac-ecab33aa21ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=1578277a38e84d42b5c56718e895f251)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:20.715308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee3a25a-e97e-4166-afac-ecab33aa21ac admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:20.729958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f9d5b7-f7c4-48b2-8086-8369032d55c9 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:20.740342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f8da8b-e546-4ca7-886e-7c2f4cb99a78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:20.741438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f8da8b-e546-4ca7-886e-7c2f4cb99a78 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:20.741888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f8da8b-e546-4ca7-886e-7c2f4cb99a78 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:20.742333 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f8da8b-e546-4ca7-886e-7c2f4cb99a78 admin admin] PATH_INFO: `/v3/projects/5c02f855b54c41f1a0c859eed726d0ed/users/520331d6b72a408491fa1bf4eb9a0b13/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:20.746383 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f8da8b-e546-4ca7-886e-7c2f4cb99a78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c02f855b54c41f1a0c859eed726d0ed, user_id=520331d6b72a408491fa1bf4eb9a0b13, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:20.752563 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f8da8b-e546-4ca7-886e-7c2f4cb99a78 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:20.785178 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1138/2272] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:20 2022] PUT /identity/v3/projects/5c02f855b54c41f1a0c859eed726d0ed/users/520331d6b72a408491fa1bf4eb9a0b13/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:20.803775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea280493-6ebb-4f7a-914e-47ab21aa42b0 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:20.804135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea280493-6ebb-4f7a-914e-47ab21aa42b0 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:20.804635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea280493-6ebb-4f7a-914e-47ab21aa42b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:20.813384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8ee3a25a-e97e-4166-afac-ecab33aa21ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1578277a38e84d42b5c56718e895f251'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:20.832996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8ee3a25a-e97e-4166-afac-ecab33aa21ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1578277a38e84d42b5c56718e895f251'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:20.834288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8ee3a25a-e97e-4166-afac-ecab33aa21ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1578277a38e84d42b5c56718e895f251'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:20.866282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8ee3a25a-e97e-4166-afac-ecab33aa21ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1578277a38e84d42b5c56718e895f251'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:20.867554 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8ee3a25a-e97e-4166-afac-ecab33aa21ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1578277a38e84d42b5c56718e895f251'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:20.892163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ea280493-6ebb-4f7a-914e-47ab21aa42b0 None None] MFA Rules not processed for user `520331d6b72a408491fa1bf4eb9a0b13`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:20.896153 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ea280493-6ebb-4f7a-914e-47ab21aa42b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:20.903184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1135/2273] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:10:20 2022] DELETE /identity/v3/users/1578277a38e84d42b5c56718e895f251 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:20.909934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420f758e-1933-4952-903a-5e4f2f00b96b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:20.910607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420f758e-1933-4952-903a-5e4f2f00b96b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:20.911771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420f758e-1933-4952-903a-5e4f2f00b96b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:20.925107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:20.953910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-420f758e-1933-4952-903a-5e4f2f00b96b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:20.956275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-420f758e-1933-4952-903a-5e4f2f00b96b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:21.048482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1139/2274] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:20 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:21.060658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea280493-6ebb-4f7a-914e-47ab21aa42b0 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:21.062814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ea280493-6ebb-4f7a-914e-47ab21aa42b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:21.077879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1136/2275] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:21.086563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420f758e-1933-4952-903a-5e4f2f00b96b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:21.088955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-420f758e-1933-4952-903a-5e4f2f00b96b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:21.119795 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:21.191476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea280493-6ebb-4f7a-914e-47ab21aa42b0 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:21.262375 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420f758e-1933-4952-903a-5e4f2f00b96b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:21.272120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476bac4c-dfb4-4070-9ff7-97e0cb7a5fb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:21.272775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476bac4c-dfb4-4070-9ff7-97e0cb7a5fb7 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:21.273055 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476bac4c-dfb4-4070-9ff7-97e0cb7a5fb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:21.273932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476bac4c-dfb4-4070-9ff7-97e0cb7a5fb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:21.298362 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476bac4c-dfb4-4070-9ff7-97e0cb7a5fb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:21.300612 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476bac4c-dfb4-4070-9ff7-97e0cb7a5fb7 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:21.336405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1137/2276] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:21.402989 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5adfe5ea-fd82-429e-8ea3-17db0d5676f8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:21.404206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adfe5ea-fd82-429e-8ea3-17db0d5676f8 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:21.405984 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adfe5ea-fd82-429e-8ea3-17db0d5676f8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:21.407624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adfe5ea-fd82-429e-8ea3-17db0d5676f8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:21.653191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5adfe5ea-fd82-429e-8ea3-17db0d5676f8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:21.657170 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5adfe5ea-fd82-429e-8ea3-17db0d5676f8 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:21.688148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1140/2277] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:21 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 628 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:10:21.910754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476bac4c-dfb4-4070-9ff7-97e0cb7a5fb7 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:21.912559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-476bac4c-dfb4-4070-9ff7-97e0cb7a5fb7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:21.940445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476bac4c-dfb4-4070-9ff7-97e0cb7a5fb7 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:21.949080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60c3505-1e50-47d0-b089-dc547069dc91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:21.950172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60c3505-1e50-47d0-b089-dc547069dc91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:21.950660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60c3505-1e50-47d0-b089-dc547069dc91 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:21.951792 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60c3505-1e50-47d0-b089-dc547069dc91 admin admin] PATH_INFO: `/v3/projects/8928680faee44bf4a612a289e3c79665` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:21.954344 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60c3505-1e50-47d0-b089-dc547069dc91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8928680faee44bf4a612a289e3c79665)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:21.957836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60c3505-1e50-47d0-b089-dc547069dc91 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:22.046361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a60c3505-1e50-47d0-b089-dc547069dc91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8928680faee44bf4a612a289e3c79665', 'request_id': 'req-a60c3505-1e50-47d0-b089-dc547069dc91'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:22.057231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a60c3505-1e50-47d0-b089-dc547069dc91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8928680faee44bf4a612a289e3c79665', 'request_id': 'req-a60c3505-1e50-47d0-b089-dc547069dc91'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:22.066326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a60c3505-1e50-47d0-b089-dc547069dc91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8928680faee44bf4a612a289e3c79665', 'request_id': 'req-a60c3505-1e50-47d0-b089-dc547069dc91'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:22.087879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a60c3505-1e50-47d0-b089-dc547069dc91 admin admin] The token cache is being invalidate because project 8928680faee44bf4a612a289e3c79665 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:22.088296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a60c3505-1e50-47d0-b089-dc547069dc91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:22.089383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a60c3505-1e50-47d0-b089-dc547069dc91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:22.092286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1138/2278] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:21 2022] DELETE /identity/v3/projects/8928680faee44bf4a612a289e3c79665 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:10:22.960610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5adfe5ea-fd82-429e-8ea3-17db0d5676f8 service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:22.963149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5adfe5ea-fd82-429e-8ea3-17db0d5676f8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:22.990671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5adfe5ea-fd82-429e-8ea3-17db0d5676f8 service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:23.016787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:23.132061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:23.132711 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:23.133127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:23.133463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] PATH_INFO: `/v3/users/d794705f3112451abe273ac7265ad053` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:23.135615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] RBAC: Authorizing `identity:delete_user(user_id=d794705f3112451abe273ac7265ad053)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:23.138033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:23.219119 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd794705f3112451abe273ac7265ad053'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:23.220046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd794705f3112451abe273ac7265ad053'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:23.230950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd794705f3112451abe273ac7265ad053'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:23.256478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd794705f3112451abe273ac7265ad053'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:23.275989 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd794705f3112451abe273ac7265ad053'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:23.284310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1141/2279] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:10:22 2022] DELETE /identity/v3/users/d794705f3112451abe273ac7265ad053 => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:23.289455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae26a34d-be2d-453a-8a36-93b0ea05fa5c None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:23.290162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae26a34d-be2d-453a-8a36-93b0ea05fa5c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:23.290162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae26a34d-be2d-453a-8a36-93b0ea05fa5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:23.313805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ae26a34d-be2d-453a-8a36-93b0ea05fa5c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:23.315428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ae26a34d-be2d-453a-8a36-93b0ea05fa5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:23.403647 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1139/2280] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:23.417561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:23.422727 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:23.564417 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a07fe6-52fa-4cdb-a0e4-303aaa1f259e admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:23.570878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c18311-ebd7-4ec5-bfe8-00dde8e6950d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:23.571919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c18311-ebd7-4ec5-bfe8-00dde8e6950d service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:23.572430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c18311-ebd7-4ec5-bfe8-00dde8e6950d service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:23.572931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c18311-ebd7-4ec5-bfe8-00dde8e6950d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:23.591195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c18311-ebd7-4ec5-bfe8-00dde8e6950d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:23.594719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c18311-ebd7-4ec5-bfe8-00dde8e6950d service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:23.614186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1142/2281] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:24.052957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae26a34d-be2d-453a-8a36-93b0ea05fa5c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:24.055295 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ae26a34d-be2d-453a-8a36-93b0ea05fa5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:24.085593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae26a34d-be2d-453a-8a36-93b0ea05fa5c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:24.097985 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:24.099641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:24.100456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:24.100949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] PATH_INFO: `/v3/projects/6e84f8b62a0e484db45b816e65055e8f` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:24.103597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e84f8b62a0e484db45b816e65055e8f)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:24.107746 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:24.186316 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:24.217244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e84f8b62a0e484db45b816e65055e8f', 'request_id': 'req-12734e09-2bc2-4f8a-822d-62965d836ab5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:24.232559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e84f8b62a0e484db45b816e65055e8f', 'request_id': 'req-12734e09-2bc2-4f8a-822d-62965d836ab5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:24.243156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e84f8b62a0e484db45b816e65055e8f', 'request_id': 'req-12734e09-2bc2-4f8a-822d-62965d836ab5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:24.274599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] The token cache is being invalidate because project 6e84f8b62a0e484db45b816e65055e8f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:24.275517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:24.276805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:24.279764 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1140/2282] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:24 2022] DELETE /identity/v3/projects/6e84f8b62a0e484db45b816e65055e8f => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:24.286225 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c18311-ebd7-4ec5-bfe8-00dde8e6950d service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:24.313652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c18311-ebd7-4ec5-bfe8-00dde8e6950d service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:24.401604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:24.466534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362b15c1-ff55-41cc-8d21-d1ff50b8b3ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:24.468610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362b15c1-ff55-41cc-8d21-d1ff50b8b3ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:24.469120 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362b15c1-ff55-41cc-8d21-d1ff50b8b3ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:24.469672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362b15c1-ff55-41cc-8d21-d1ff50b8b3ef admin admin] PATH_INFO: `/v3/users/520331d6b72a408491fa1bf4eb9a0b13` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:24.472342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362b15c1-ff55-41cc-8d21-d1ff50b8b3ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=520331d6b72a408491fa1bf4eb9a0b13)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:24.476506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362b15c1-ff55-41cc-8d21-d1ff50b8b3ef admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:24.611567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-362b15c1-ff55-41cc-8d21-d1ff50b8b3ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '520331d6b72a408491fa1bf4eb9a0b13'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:24.612349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-362b15c1-ff55-41cc-8d21-d1ff50b8b3ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '520331d6b72a408491fa1bf4eb9a0b13'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:24.628095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-362b15c1-ff55-41cc-8d21-d1ff50b8b3ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '520331d6b72a408491fa1bf4eb9a0b13'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:24.659960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-362b15c1-ff55-41cc-8d21-d1ff50b8b3ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '520331d6b72a408491fa1bf4eb9a0b13'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:24.680944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-362b15c1-ff55-41cc-8d21-d1ff50b8b3ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '520331d6b72a408491fa1bf4eb9a0b13'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:24.710943 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1143/2283] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:10:24 2022] DELETE /identity/v3/users/520331d6b72a408491fa1bf4eb9a0b13 => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:25.155669 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:25.161444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:25.208993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12734e09-2bc2-4f8a-822d-62965d836ab5 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:25.338288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9876ff7e-a5b0-4384-bbdd-72c83960c318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:25.339616 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9876ff7e-a5b0-4384-bbdd-72c83960c318 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:25.340206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9876ff7e-a5b0-4384-bbdd-72c83960c318 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:25.340831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9876ff7e-a5b0-4384-bbdd-72c83960c318 admin admin] PATH_INFO: `/v3/projects/5c02f855b54c41f1a0c859eed726d0ed` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:25.343806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9876ff7e-a5b0-4384-bbdd-72c83960c318 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c02f855b54c41f1a0c859eed726d0ed)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:25.347757 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9876ff7e-a5b0-4384-bbdd-72c83960c318 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:25.447262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9876ff7e-a5b0-4384-bbdd-72c83960c318 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c02f855b54c41f1a0c859eed726d0ed', 'request_id': 'req-9876ff7e-a5b0-4384-bbdd-72c83960c318'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:25.479917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9876ff7e-a5b0-4384-bbdd-72c83960c318 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c02f855b54c41f1a0c859eed726d0ed', 'request_id': 'req-9876ff7e-a5b0-4384-bbdd-72c83960c318'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:25.509174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9876ff7e-a5b0-4384-bbdd-72c83960c318 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c02f855b54c41f1a0c859eed726d0ed', 'request_id': 'req-9876ff7e-a5b0-4384-bbdd-72c83960c318'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:25.595288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9876ff7e-a5b0-4384-bbdd-72c83960c318 admin admin] The token cache is being invalidate because project 5c02f855b54c41f1a0c859eed726d0ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:25.597957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9876ff7e-a5b0-4384-bbdd-72c83960c318 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:25.602250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9876ff7e-a5b0-4384-bbdd-72c83960c318 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:25.612904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1141/2284] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:25 2022] DELETE /identity/v3/projects/5c02f855b54c41f1a0c859eed726d0ed => generated 0 bytes in 460 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:25.620513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6789eef-7549-4785-9fb2-b125234149a7 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:25.621893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6789eef-7549-4785-9fb2-b125234149a7 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:25.624289 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6789eef-7549-4785-9fb2-b125234149a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:25.673644 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-e6789eef-7549-4785-9fb2-b125234149a7 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:25.681940 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e6789eef-7549-4785-9fb2-b125234149a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:25.771982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1144/2285] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:25.779750 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cbbccc-0e99-4006-a3c6-d0b68eaec8ce None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:25.780278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cbbccc-0e99-4006-a3c6-d0b68eaec8ce None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:25.780843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cbbccc-0e99-4006-a3c6-d0b68eaec8ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:25.821110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-27cbbccc-0e99-4006-a3c6-d0b68eaec8ce None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:25.830407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-27cbbccc-0e99-4006-a3c6-d0b68eaec8ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:25.868753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1142/2286] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:25.874765 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6789eef-7549-4785-9fb2-b125234149a7 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:25.890034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6789eef-7549-4785-9fb2-b125234149a7 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:25.903412 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36104ab6-0f1d-49c9-888e-3809e9d79c54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:25.904346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36104ab6-0f1d-49c9-888e-3809e9d79c54 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:25.905045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36104ab6-0f1d-49c9-888e-3809e9d79c54 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:25.908501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36104ab6-0f1d-49c9-888e-3809e9d79c54 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:25.909627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36104ab6-0f1d-49c9-888e-3809e9d79c54 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:10:25.910213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36104ab6-0f1d-49c9-888e-3809e9d79c54 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:25.915032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36104ab6-0f1d-49c9-888e-3809e9d79c54 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:25.938882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1145/2287] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:10:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:10:25.945406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cbbccc-0e99-4006-a3c6-d0b68eaec8ce None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:25.964759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cbbccc-0e99-4006-a3c6-d0b68eaec8ce None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:25.979063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd32f585-1e39-4b4d-a38c-3dbcc2af0f84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:25.979875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd32f585-1e39-4b4d-a38c-3dbcc2af0f84 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:25.980161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd32f585-1e39-4b4d-a38c-3dbcc2af0f84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:25.980435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd32f585-1e39-4b4d-a38c-3dbcc2af0f84 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:25.981151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd32f585-1e39-4b4d-a38c-3dbcc2af0f84 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:25.999229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd32f585-1e39-4b4d-a38c-3dbcc2af0f84 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:26.081858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1143/2288] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:10:25 2022] POST /identity/v3/projects => generated 375 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:10:26.087901 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36104ab6-0f1d-49c9-888e-3809e9d79c54 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:26.102314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36104ab6-0f1d-49c9-888e-3809e9d79c54 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:26.191912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:26.241629 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc12cf9-8cc9-4787-a3d1-198a500d2146 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:26.242701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc12cf9-8cc9-4787-a3d1-198a500d2146 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:26.243004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc12cf9-8cc9-4787-a3d1-198a500d2146 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:26.243538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc12cf9-8cc9-4787-a3d1-198a500d2146 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:26.244517 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc12cf9-8cc9-4787-a3d1-198a500d2146 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:26.248869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc12cf9-8cc9-4787-a3d1-198a500d2146 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:26.291470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1146/2289] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:10:26 2022] POST /identity/v3/users => generated 288 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:10:26.296201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd32f585-1e39-4b4d-a38c-3dbcc2af0f84 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:26.317654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd32f585-1e39-4b4d-a38c-3dbcc2af0f84 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:26.325186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09482dab-8c33-4755-8762-99cbe92c12d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:26.326315 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09482dab-8c33-4755-8762-99cbe92c12d7 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:26.326777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09482dab-8c33-4755-8762-99cbe92c12d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:26.328190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09482dab-8c33-4755-8762-99cbe92c12d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:26.329726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09482dab-8c33-4755-8762-99cbe92c12d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:26.333222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09482dab-8c33-4755-8762-99cbe92c12d7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:26.366762 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1144/2290] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:26 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:26.367346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc12cf9-8cc9-4787-a3d1-198a500d2146 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:26.386597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc12cf9-8cc9-4787-a3d1-198a500d2146 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:26.407238 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6701a6c6-ba0d-43c1-8831-9a8864d4c612 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:26.408043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6701a6c6-ba0d-43c1-8831-9a8864d4c612 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:26.408339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6701a6c6-ba0d-43c1-8831-9a8864d4c612 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:26.408717 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6701a6c6-ba0d-43c1-8831-9a8864d4c612 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:26.409532 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6701a6c6-ba0d-43c1-8831-9a8864d4c612 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:26.411704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6701a6c6-ba0d-43c1-8831-9a8864d4c612 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:26.432198 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1147/2291] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:26 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:10:26.439523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09482dab-8c33-4755-8762-99cbe92c12d7 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:26.458367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09482dab-8c33-4755-8762-99cbe92c12d7 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:26.467333 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a13c9ad-4548-465f-a991-f59f981158dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:26.468766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a13c9ad-4548-465f-a991-f59f981158dd admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:26.469530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a13c9ad-4548-465f-a991-f59f981158dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:26.470118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a13c9ad-4548-465f-a991-f59f981158dd admin admin] PATH_INFO: `/v3/projects/cacc8e9a98d94c72be4d3f28942e5ba3/users/0b82e65b31044b42ba0a2d05f391219c/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:26.498274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:26.504067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a13c9ad-4548-465f-a991-f59f981158dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=cacc8e9a98d94c72be4d3f28942e5ba3, user_id=0b82e65b31044b42ba0a2d05f391219c, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:26.509675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a13c9ad-4548-465f-a991-f59f981158dd admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:26.539646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1145/2292] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:26 2022] PUT /identity/v3/projects/cacc8e9a98d94c72be4d3f28942e5ba3/users/0b82e65b31044b42ba0a2d05f391219c/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:26.547668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90e30fb-c993-4bb9-9eb7-897932530304 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:26.548006 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90e30fb-c993-4bb9-9eb7-897932530304 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:26.548347 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90e30fb-c993-4bb9-9eb7-897932530304 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:26.606123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-c90e30fb-c993-4bb9-9eb7-897932530304 None None] MFA Rules not processed for user `0b82e65b31044b42ba0a2d05f391219c`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:26.607787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c90e30fb-c993-4bb9-9eb7-897932530304 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:26.714678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1148/2293] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:26 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:26.729440 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a13c9ad-4548-465f-a991-f59f981158dd admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:26.731583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6a13c9ad-4548-465f-a991-f59f981158dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:26.835093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a13c9ad-4548-465f-a991-f59f981158dd admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:26.839061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3112e55-6261-4702-a429-d485fd344b23 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4160039561434e51a87c6c7a019f2dff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:26.839747 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3112e55-6261-4702-a429-d485fd344b23 service glance] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:26.840025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3112e55-6261-4702-a429-d485fd344b23 service glance] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:26.840300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3112e55-6261-4702-a429-d485fd344b23 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:26.850715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3112e55-6261-4702-a429-d485fd344b23 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:26.852324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3112e55-6261-4702-a429-d485fd344b23 service glance] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:26.865982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1146/2294] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:26 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:27.345531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90e30fb-c993-4bb9-9eb7-897932530304 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:27.348567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c90e30fb-c993-4bb9-9eb7-897932530304 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:27.571036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90e30fb-c993-4bb9-9eb7-897932530304 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:27.585482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:27.586190 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:27.586467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:27.586739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] PATH_INFO: `/v3/users/6e733490ff584013956a16c2611008e3` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:27.588136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e733490ff584013956a16c2611008e3)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:27.590160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:27.714875 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e733490ff584013956a16c2611008e3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:27.715954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e733490ff584013956a16c2611008e3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:27.728637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e733490ff584013956a16c2611008e3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:27.730458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:27.759648 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e733490ff584013956a16c2611008e3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:27.778741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e733490ff584013956a16c2611008e3'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:27.795881 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1149/2295] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:10:27 2022] DELETE /identity/v3/users/6e733490ff584013956a16c2611008e3 => generated 0 bytes in 448 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:27.797707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add7df39-c53b-46e7-9ebc-03ce9016da51 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:27.798205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add7df39-c53b-46e7-9ebc-03ce9016da51 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:27.798347 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add7df39-c53b-46e7-9ebc-03ce9016da51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:27.886920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-add7df39-c53b-46e7-9ebc-03ce9016da51 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:27.898292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-add7df39-c53b-46e7-9ebc-03ce9016da51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:28.147831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1147/2296] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 351 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:28.212225 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:28.213950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:28.461940 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a8289f-f328-4ae1-ac6c-6f0cf7951874 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:28.469245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4065f145-e72e-477e-a00b-3bf47c5647f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:28.470354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4065f145-e72e-477e-a00b-3bf47c5647f3 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:28.470933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4065f145-e72e-477e-a00b-3bf47c5647f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:28.471484 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4065f145-e72e-477e-a00b-3bf47c5647f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:28.499907 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4065f145-e72e-477e-a00b-3bf47c5647f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:28.502698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4065f145-e72e-477e-a00b-3bf47c5647f3 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:28.540414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1150/2297] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:10:29.404557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add7df39-c53b-46e7-9ebc-03ce9016da51 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:29.407274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-add7df39-c53b-46e7-9ebc-03ce9016da51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:29.434484 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add7df39-c53b-46e7-9ebc-03ce9016da51 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:29.443397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:29.444542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:29.445041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:29.445415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579 admin admin] PATH_INFO: `/v3/projects/e4f21990c9ae4a2a9b9ca2c5680d9a1b` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:29.447295 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4f21990c9ae4a2a9b9ca2c5680d9a1b)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:29.449508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:29.538506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4f21990c9ae4a2a9b9ca2c5680d9a1b', 'request_id': 'req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:29.554781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4f21990c9ae4a2a9b9ca2c5680d9a1b', 'request_id': 'req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:29.574225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4f21990c9ae4a2a9b9ca2c5680d9a1b', 'request_id': 'req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:29.579964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:29.601827 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579 admin admin] The token cache is being invalidate because project e4f21990c9ae4a2a9b9ca2c5680d9a1b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:29.602286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:29.603690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-94cc9cc3-5d34-4bfc-b9f8-8945617a1579 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:29.607330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1148/2298] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:29 2022] DELETE /identity/v3/projects/e4f21990c9ae4a2a9b9ca2c5680d9a1b => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:10:29.615260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d04a72d-4cbb-4d17-8abc-08283a987039 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:29.616185 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d04a72d-4cbb-4d17-8abc-08283a987039 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:29.616502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d04a72d-4cbb-4d17-8abc-08283a987039 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:29.643493 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-2d04a72d-4cbb-4d17-8abc-08283a987039 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:29.644793 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2d04a72d-4cbb-4d17-8abc-08283a987039 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:29.733444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1151/2299] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:29.743306 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f784964-a191-4c23-8417-50174cab1073 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:29.743863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f784964-a191-4c23-8417-50174cab1073 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:29.744411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f784964-a191-4c23-8417-50174cab1073 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:29.779547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-9f784964-a191-4c23-8417-50174cab1073 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:29.781845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9f784964-a191-4c23-8417-50174cab1073 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:29.796910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1149/2300] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:29.801408 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d04a72d-4cbb-4d17-8abc-08283a987039 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:29.813304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d04a72d-4cbb-4d17-8abc-08283a987039 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:29.818543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720c793-d864-46e1-9412-d873ee60952d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:29.819180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720c793-d864-46e1-9412-d873ee60952d admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:29.819501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720c793-d864-46e1-9412-d873ee60952d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:29.819772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720c793-d864-46e1-9412-d873ee60952d admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:29.820539 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720c793-d864-46e1-9412-d873ee60952d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:10:29.820843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720c793-d864-46e1-9412-d873ee60952d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:29.822523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720c793-d864-46e1-9412-d873ee60952d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:29.837059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1152/2301] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:10:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:10:29.844359 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f784964-a191-4c23-8417-50174cab1073 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:29.859669 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f784964-a191-4c23-8417-50174cab1073 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:29.864910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fe8dc8-49f7-40eb-aa5e-894703916443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:29.865545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fe8dc8-49f7-40eb-aa5e-894703916443 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:29.865825 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fe8dc8-49f7-40eb-aa5e-894703916443 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:29.866095 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fe8dc8-49f7-40eb-aa5e-894703916443 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:29.866797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fe8dc8-49f7-40eb-aa5e-894703916443 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:29.868895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fe8dc8-49f7-40eb-aa5e-894703916443 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:29.917589 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1150/2302] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:10:29 2022] POST /identity/v3/projects => generated 375 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:10:29.920536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720c793-d864-46e1-9412-d873ee60952d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:29.935132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720c793-d864-46e1-9412-d873ee60952d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:30.012919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b522dc-5bf1-474d-a0cc-b002c69067a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:30.014267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b522dc-5bf1-474d-a0cc-b002c69067a2 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:30.014886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b522dc-5bf1-474d-a0cc-b002c69067a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:30.015393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b522dc-5bf1-474d-a0cc-b002c69067a2 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:30.016697 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b522dc-5bf1-474d-a0cc-b002c69067a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:30.019815 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b522dc-5bf1-474d-a0cc-b002c69067a2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:30.031042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:30.050784 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1153/2303] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:10:29 2022] POST /identity/v3/users => generated 288 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:10:30.053381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fe8dc8-49f7-40eb-aa5e-894703916443 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:30.073348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fe8dc8-49f7-40eb-aa5e-894703916443 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:30.080580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b740c2d-ee42-4b07-b8d3-ac040a82f4f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:30.081562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b740c2d-ee42-4b07-b8d3-ac040a82f4f7 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:30.082116 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b740c2d-ee42-4b07-b8d3-ac040a82f4f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:30.082488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b740c2d-ee42-4b07-b8d3-ac040a82f4f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:30.083703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b740c2d-ee42-4b07-b8d3-ac040a82f4f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:30.085488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b740c2d-ee42-4b07-b8d3-ac040a82f4f7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:30.097758 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1151/2304] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:30 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:30.102168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b522dc-5bf1-474d-a0cc-b002c69067a2 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:30.113836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b522dc-5bf1-474d-a0cc-b002c69067a2 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:30.119164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ebaae7-ecaf-41bb-ba1d-7562796b9d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:30.119864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ebaae7-ecaf-41bb-ba1d-7562796b9d4a admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:30.120168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ebaae7-ecaf-41bb-ba1d-7562796b9d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:30.120444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ebaae7-ecaf-41bb-ba1d-7562796b9d4a admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:30.121168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ebaae7-ecaf-41bb-ba1d-7562796b9d4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:30.122971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ebaae7-ecaf-41bb-ba1d-7562796b9d4a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:30.136388 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1154/2305] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:10:30.143400 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b740c2d-ee42-4b07-b8d3-ac040a82f4f7 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:30.163874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b740c2d-ee42-4b07-b8d3-ac040a82f4f7 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:30.180448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aee2863-f904-4f43-9de8-587c8ec2aa17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:30.185150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aee2863-f904-4f43-9de8-587c8ec2aa17 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:30.185765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aee2863-f904-4f43-9de8-587c8ec2aa17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:30.186263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aee2863-f904-4f43-9de8-587c8ec2aa17 admin admin] PATH_INFO: `/v3/projects/cd9357230283487bbd756edae65479f7/users/41dd8d5a965b43559568bf0f09031b6d/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:30.226376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aee2863-f904-4f43-9de8-587c8ec2aa17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd9357230283487bbd756edae65479f7, user_id=41dd8d5a965b43559568bf0f09031b6d, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:30.236403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aee2863-f904-4f43-9de8-587c8ec2aa17 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:30.277227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1152/2306] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:30 2022] PUT /identity/v3/projects/cd9357230283487bbd756edae65479f7/users/41dd8d5a965b43559568bf0f09031b6d/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:30.316536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca2fd7a-ac54-4ff9-ae85-da6c2dc771ed None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:30.319041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca2fd7a-ac54-4ff9-ae85-da6c2dc771ed None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:30.320218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca2fd7a-ac54-4ff9-ae85-da6c2dc771ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:30.424208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-aca2fd7a-ac54-4ff9-ae85-da6c2dc771ed None None] MFA Rules not processed for user `41dd8d5a965b43559568bf0f09031b6d`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:30.435478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-aca2fd7a-ac54-4ff9-ae85-da6c2dc771ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:30.640662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1155/2307] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:30 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 329 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:30.666431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aee2863-f904-4f43-9de8-587c8ec2aa17 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:30.671625 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8aee2863-f904-4f43-9de8-587c8ec2aa17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:30.813390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aee2863-f904-4f43-9de8-587c8ec2aa17 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:30.820745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a8343a-a120-47c1-bb79-cf918b8e3065 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:30.822017 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a8343a-a120-47c1-bb79-cf918b8e3065 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:30.822633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a8343a-a120-47c1-bb79-cf918b8e3065 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:30.823257 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a8343a-a120-47c1-bb79-cf918b8e3065 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:30.843521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a8343a-a120-47c1-bb79-cf918b8e3065 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:30.846408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a8343a-a120-47c1-bb79-cf918b8e3065 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:30.868110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1153/2308] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:30 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:35.948339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca2fd7a-ac54-4ff9-ae85-da6c2dc771ed None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:36.041632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca2fd7a-ac54-4ff9-ae85-da6c2dc771ed None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:36.049268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6edd905-4dc0-4fe6-98e4-59af003a78e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:36.050369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6edd905-4dc0-4fe6-98e4-59af003a78e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:36.050719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6edd905-4dc0-4fe6-98e4-59af003a78e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:36.051171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6edd905-4dc0-4fe6-98e4-59af003a78e4 admin admin] PATH_INFO: `/v3/users/41dd8d5a965b43559568bf0f09031b6d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:36.053357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6edd905-4dc0-4fe6-98e4-59af003a78e4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=41dd8d5a965b43559568bf0f09031b6d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:36.057011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6edd905-4dc0-4fe6-98e4-59af003a78e4 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:36.083826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a8343a-a120-47c1-bb79-cf918b8e3065 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:36.086325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-93a8343a-a120-47c1-bb79-cf918b8e3065 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:36.113645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a8343a-a120-47c1-bb79-cf918b8e3065 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:36.131160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c095e7d8-f20b-43a7-9cb2-76b96fd8910b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:36.133614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c095e7d8-f20b-43a7-9cb2-76b96fd8910b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:36.134219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c095e7d8-f20b-43a7-9cb2-76b96fd8910b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:36.134752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c095e7d8-f20b-43a7-9cb2-76b96fd8910b admin admin] PATH_INFO: `/v3/users/6febcd137bf44b9b8cd32deec0896f74` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:36.137109 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c095e7d8-f20b-43a7-9cb2-76b96fd8910b admin admin] RBAC: Authorizing `identity:delete_user(user_id=6febcd137bf44b9b8cd32deec0896f74)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:36.141282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c095e7d8-f20b-43a7-9cb2-76b96fd8910b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:36.162087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f6edd905-4dc0-4fe6-98e4-59af003a78e4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '41dd8d5a965b43559568bf0f09031b6d'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:36.164095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f6edd905-4dc0-4fe6-98e4-59af003a78e4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '41dd8d5a965b43559568bf0f09031b6d'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:36.180479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f6edd905-4dc0-4fe6-98e4-59af003a78e4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '41dd8d5a965b43559568bf0f09031b6d'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:36.220637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f6edd905-4dc0-4fe6-98e4-59af003a78e4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '41dd8d5a965b43559568bf0f09031b6d'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:36.224072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:INSERT=3,keystone:SELECT=87,keystone:DELETE=10 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:36.240929 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f6edd905-4dc0-4fe6-98e4-59af003a78e4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '41dd8d5a965b43559568bf0f09031b6d'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:36.257702 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1156/2309] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:10:35 2022] DELETE /identity/v3/users/41dd8d5a965b43559568bf0f09031b6d => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:36.265826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa1448a-95eb-47ac-8ac1-80c9a327b41e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:36.267479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa1448a-95eb-47ac-8ac1-80c9a327b41e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:36.268981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa1448a-95eb-47ac-8ac1-80c9a327b41e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:36.293664 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c095e7d8-f20b-43a7-9cb2-76b96fd8910b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6febcd137bf44b9b8cd32deec0896f74'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:36.307628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-eaa1448a-95eb-47ac-8ac1-80c9a327b41e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:36.311809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-eaa1448a-95eb-47ac-8ac1-80c9a327b41e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:36.316431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c095e7d8-f20b-43a7-9cb2-76b96fd8910b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6febcd137bf44b9b8cd32deec0896f74'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:36.317532 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c095e7d8-f20b-43a7-9cb2-76b96fd8910b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6febcd137bf44b9b8cd32deec0896f74'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:36.362905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c095e7d8-f20b-43a7-9cb2-76b96fd8910b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6febcd137bf44b9b8cd32deec0896f74'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:36.376342 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c095e7d8-f20b-43a7-9cb2-76b96fd8910b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6febcd137bf44b9b8cd32deec0896f74'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:36.398431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1154/2310] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:10:36 2022] DELETE /identity/v3/users/6febcd137bf44b9b8cd32deec0896f74 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:36.407786 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f94af2a-1939-452a-9a44-b2f07cf37ea4 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:36.408411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f94af2a-1939-452a-9a44-b2f07cf37ea4 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:36.409051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f94af2a-1939-452a-9a44-b2f07cf37ea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:36.432311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1157/2311] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:36.437035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:36.443997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa1448a-95eb-47ac-8ac1-80c9a327b41e None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:36.446560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-eaa1448a-95eb-47ac-8ac1-80c9a327b41e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:36.461840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6f94af2a-1939-452a-9a44-b2f07cf37ea4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:36.465190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6f94af2a-1939-452a-9a44-b2f07cf37ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:36.576283 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa1448a-95eb-47ac-8ac1-80c9a327b41e None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:36.595458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1155/2312] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:36.608233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f94af2a-1939-452a-9a44-b2f07cf37ea4 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:36.686257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764d295c-963b-4e3f-b53b-c9adcfbf57a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:36.687977 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764d295c-963b-4e3f-b53b-c9adcfbf57a9 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:36.688453 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764d295c-963b-4e3f-b53b-c9adcfbf57a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:36.689237 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764d295c-963b-4e3f-b53b-c9adcfbf57a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:36.712955 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764d295c-963b-4e3f-b53b-c9adcfbf57a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:36.714932 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764d295c-963b-4e3f-b53b-c9adcfbf57a9 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:36.715702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f94af2a-1939-452a-9a44-b2f07cf37ea4 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:36.720829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc768216-bc0a-468d-9ee9-5808cb3cceff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:36.721683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc768216-bc0a-468d-9ee9-5808cb3cceff service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:36.722263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc768216-bc0a-468d-9ee9-5808cb3cceff service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:36.722488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc768216-bc0a-468d-9ee9-5808cb3cceff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:36.730574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1158/2313] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:10:36.735669 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc768216-bc0a-468d-9ee9-5808cb3cceff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:36.737272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc768216-bc0a-468d-9ee9-5808cb3cceff service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:36.751002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1156/2314] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:37.257859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764d295c-963b-4e3f-b53b-c9adcfbf57a9 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:37.260992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-764d295c-963b-4e3f-b53b-c9adcfbf57a9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:37.273979 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764d295c-963b-4e3f-b53b-c9adcfbf57a9 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:37.286496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7beeb56-f270-4377-9096-366b1c8a126e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:37.287176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7beeb56-f270-4377-9096-366b1c8a126e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:37.287495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7beeb56-f270-4377-9096-366b1c8a126e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:37.287778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7beeb56-f270-4377-9096-366b1c8a126e admin admin] PATH_INFO: `/v3/projects/cd9357230283487bbd756edae65479f7` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:37.289268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7beeb56-f270-4377-9096-366b1c8a126e admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd9357230283487bbd756edae65479f7)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:37.291310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7beeb56-f270-4377-9096-366b1c8a126e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:37.350474 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:37.356509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e7beeb56-f270-4377-9096-366b1c8a126e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd9357230283487bbd756edae65479f7', 'request_id': 'req-e7beeb56-f270-4377-9096-366b1c8a126e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:37.365824 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e7beeb56-f270-4377-9096-366b1c8a126e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd9357230283487bbd756edae65479f7', 'request_id': 'req-e7beeb56-f270-4377-9096-366b1c8a126e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:37.377985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e7beeb56-f270-4377-9096-366b1c8a126e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd9357230283487bbd756edae65479f7', 'request_id': 'req-e7beeb56-f270-4377-9096-366b1c8a126e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:37.396710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e7beeb56-f270-4377-9096-366b1c8a126e admin admin] The token cache is being invalidate because project cd9357230283487bbd756edae65479f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:37.397192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e7beeb56-f270-4377-9096-366b1c8a126e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:37.398315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e7beeb56-f270-4377-9096-366b1c8a126e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:37.401633 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1159/2315] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:37 2022] DELETE /identity/v3/projects/cd9357230283487bbd756edae65479f7 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:10:37.660274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc768216-bc0a-468d-9ee9-5808cb3cceff service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:37.669434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-dc768216-bc0a-468d-9ee9-5808cb3cceff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:37.710157 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc768216-bc0a-468d-9ee9-5808cb3cceff service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:37.845258 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3344fb-2047-4b43-abfa-5da3db7ffcae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:37.846268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3344fb-2047-4b43-abfa-5da3db7ffcae admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:37.846910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3344fb-2047-4b43-abfa-5da3db7ffcae admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:37.847391 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3344fb-2047-4b43-abfa-5da3db7ffcae admin admin] PATH_INFO: `/v3/projects/9534f49685b24d218e97834be3980c68` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:37.849334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3344fb-2047-4b43-abfa-5da3db7ffcae admin admin] RBAC: Authorizing `identity:delete_project(project_id=9534f49685b24d218e97834be3980c68)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:37.852927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3344fb-2047-4b43-abfa-5da3db7ffcae admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:37.924814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-1f3344fb-2047-4b43-abfa-5da3db7ffcae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9534f49685b24d218e97834be3980c68', 'request_id': 'req-1f3344fb-2047-4b43-abfa-5da3db7ffcae'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:37.927248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:37.936325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-1f3344fb-2047-4b43-abfa-5da3db7ffcae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9534f49685b24d218e97834be3980c68', 'request_id': 'req-1f3344fb-2047-4b43-abfa-5da3db7ffcae'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:37.945485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-1f3344fb-2047-4b43-abfa-5da3db7ffcae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9534f49685b24d218e97834be3980c68', 'request_id': 'req-1f3344fb-2047-4b43-abfa-5da3db7ffcae'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:37.965310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-1f3344fb-2047-4b43-abfa-5da3db7ffcae admin admin] The token cache is being invalidate because project 9534f49685b24d218e97834be3980c68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:37.965606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-1f3344fb-2047-4b43-abfa-5da3db7ffcae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:37.970594 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-1f3344fb-2047-4b43-abfa-5da3db7ffcae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:37.974919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1157/2316] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:37 2022] DELETE /identity/v3/projects/9534f49685b24d218e97834be3980c68 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:37.980351 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e38b26-57a4-4792-a084-4e00ec80e75d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:37.980992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e38b26-57a4-4792-a084-4e00ec80e75d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:37.982129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e38b26-57a4-4792-a084-4e00ec80e75d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:38.016928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-84e38b26-57a4-4792-a084-4e00ec80e75d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:38.019146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-84e38b26-57a4-4792-a084-4e00ec80e75d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:38.137917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1160/2317] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:38.151397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be46113-2bfe-47cc-a51a-dc0c81efd774 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:38.151859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be46113-2bfe-47cc-a51a-dc0c81efd774 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:38.152280 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be46113-2bfe-47cc-a51a-dc0c81efd774 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:38.175962 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-1be46113-2bfe-47cc-a51a-dc0c81efd774 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:38.177437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1be46113-2bfe-47cc-a51a-dc0c81efd774 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:38.186791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1158/2318] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:38.191781 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e38b26-57a4-4792-a084-4e00ec80e75d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:38.202787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e38b26-57a4-4792-a084-4e00ec80e75d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:38.209994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a6135-2f49-4b06-a236-116989dd8082 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:38.210917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a6135-2f49-4b06-a236-116989dd8082 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:38.211481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a6135-2f49-4b06-a236-116989dd8082 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:38.211965 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a6135-2f49-4b06-a236-116989dd8082 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:38.213346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78a6135-2f49-4b06-a236-116989dd8082 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:10:38.213899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78a6135-2f49-4b06-a236-116989dd8082 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:38.218019 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78a6135-2f49-4b06-a236-116989dd8082 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:38.233489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1161/2319] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:10:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:10:38.238472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be46113-2bfe-47cc-a51a-dc0c81efd774 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:38.249038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be46113-2bfe-47cc-a51a-dc0c81efd774 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:38.254192 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41459cfd-aae1-47bf-afcb-a82d94010c72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:38.254981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41459cfd-aae1-47bf-afcb-a82d94010c72 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:38.255389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41459cfd-aae1-47bf-afcb-a82d94010c72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:38.255726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41459cfd-aae1-47bf-afcb-a82d94010c72 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:38.256475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41459cfd-aae1-47bf-afcb-a82d94010c72 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:38.258812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41459cfd-aae1-47bf-afcb-a82d94010c72 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:38.295843 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1159/2320] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:10:38 2022] POST /identity/v3/projects => generated 375 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:10:38.301692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a6135-2f49-4b06-a236-116989dd8082 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:38.313752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a6135-2f49-4b06-a236-116989dd8082 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:38.385045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec44780-fe46-4419-8e28-44c962e1dee0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:38.385574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec44780-fe46-4419-8e28-44c962e1dee0 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:38.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec44780-fe46-4419-8e28-44c962e1dee0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:38.386156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec44780-fe46-4419-8e28-44c962e1dee0 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:38.386834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec44780-fe46-4419-8e28-44c962e1dee0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:38.388800 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec44780-fe46-4419-8e28-44c962e1dee0 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:38.413205 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1162/2321] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:10:38 2022] POST /identity/v3/users => generated 288 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:10:38.420443 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41459cfd-aae1-47bf-afcb-a82d94010c72 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:38.447307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41459cfd-aae1-47bf-afcb-a82d94010c72 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:38.452472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7183625-6426-451a-bd8d-3deab145434a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:38.453164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7183625-6426-451a-bd8d-3deab145434a admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:38.453540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7183625-6426-451a-bd8d-3deab145434a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:38.453864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7183625-6426-451a-bd8d-3deab145434a admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:38.454616 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7183625-6426-451a-bd8d-3deab145434a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:38.456425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7183625-6426-451a-bd8d-3deab145434a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:38.468893 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1160/2322] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:38 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:38.473334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec44780-fe46-4419-8e28-44c962e1dee0 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:38.485590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec44780-fe46-4419-8e28-44c962e1dee0 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:38.490919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400f75df-eb8b-4d1d-a5f8-22b9fb365afd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:38.491652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400f75df-eb8b-4d1d-a5f8-22b9fb365afd admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:38.491973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400f75df-eb8b-4d1d-a5f8-22b9fb365afd admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:38.492350 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400f75df-eb8b-4d1d-a5f8-22b9fb365afd admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:38.493155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400f75df-eb8b-4d1d-a5f8-22b9fb365afd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:38.494974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400f75df-eb8b-4d1d-a5f8-22b9fb365afd admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:38.510457 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1163/2323] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:38 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:10:38.515246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7183625-6426-451a-bd8d-3deab145434a admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:38.530633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7183625-6426-451a-bd8d-3deab145434a admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:38.536522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead983d6-382f-4a15-802b-77adcb80f330 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:38.537335 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead983d6-382f-4a15-802b-77adcb80f330 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:38.537723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead983d6-382f-4a15-802b-77adcb80f330 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:38.538118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead983d6-382f-4a15-802b-77adcb80f330 admin admin] PATH_INFO: `/v3/projects/bf5ca398012441cba969788634f537b8/users/2e69ff835a0247869749c889143d8275/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:38.561303 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead983d6-382f-4a15-802b-77adcb80f330 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf5ca398012441cba969788634f537b8, user_id=2e69ff835a0247869749c889143d8275, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:38.564444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead983d6-382f-4a15-802b-77adcb80f330 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:38.580648 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1161/2324] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:38 2022] PUT /identity/v3/projects/bf5ca398012441cba969788634f537b8/users/2e69ff835a0247869749c889143d8275/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:38.594726 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2276a03f-f395-4232-8ba7-d516fef18eac None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:38.595414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2276a03f-f395-4232-8ba7-d516fef18eac None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:38.595933 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2276a03f-f395-4232-8ba7-d516fef18eac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:38.651099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-2276a03f-f395-4232-8ba7-d516fef18eac None None] MFA Rules not processed for user `2e69ff835a0247869749c889143d8275`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:38.653188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2276a03f-f395-4232-8ba7-d516fef18eac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:38.701814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:38.777759 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1164/2325] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:38 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:38.790503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead983d6-382f-4a15-802b-77adcb80f330 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:38.793227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ead983d6-382f-4a15-802b-77adcb80f330 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:38.919609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead983d6-382f-4a15-802b-77adcb80f330 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:38.924523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b70d1dc-d9a0-4355-8f06-88d7a5731aa8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:38.925236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b70d1dc-d9a0-4355-8f06-88d7a5731aa8 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:38.926386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b70d1dc-d9a0-4355-8f06-88d7a5731aa8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:38.926669 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b70d1dc-d9a0-4355-8f06-88d7a5731aa8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:38.940197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b70d1dc-d9a0-4355-8f06-88d7a5731aa8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:38.944096 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b70d1dc-d9a0-4355-8f06-88d7a5731aa8 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:38.968913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1162/2326] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:38 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:41.639868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e42a158-b7a9-40db-9cbf-f842c552e68d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:41.640337 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e42a158-b7a9-40db-9cbf-f842c552e68d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:41.640816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e42a158-b7a9-40db-9cbf-f842c552e68d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:41.674365 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-6e42a158-b7a9-40db-9cbf-f842c552e68d None None] MFA Rules not processed for user `2e69ff835a0247869749c889143d8275`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:41.682935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6e42a158-b7a9-40db-9cbf-f842c552e68d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:41.701047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1165/2327] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:41 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:43.764674 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d22e0a-c243-4cc6-92f9-f0525e19062f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:43.765509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d22e0a-c243-4cc6-92f9-f0525e19062f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:43.765509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d22e0a-c243-4cc6-92f9-f0525e19062f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:43.791577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-16d22e0a-c243-4cc6-92f9-f0525e19062f None None] MFA Rules not processed for user `8fcc34c42bb64200ac5f40d770d6784b`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:43.793162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-16d22e0a-c243-4cc6-92f9-f0525e19062f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:43.894304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1163/2328] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:43 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:51.669483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=18 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:53.884575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=83,keystone:DELETE=3,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:55.165104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e42a158-b7a9-40db-9cbf-f842c552e68d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:55.167135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6e42a158-b7a9-40db-9cbf-f842c552e68d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:55.282969 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e42a158-b7a9-40db-9cbf-f842c552e68d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:55.287989 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:55.288652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:55.288930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:55.289202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] PATH_INFO: `/v3/users/0b82e65b31044b42ba0a2d05f391219c` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:55.290510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b82e65b31044b42ba0a2d05f391219c)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:55.292511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:55.374886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b82e65b31044b42ba0a2d05f391219c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:55.376195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b82e65b31044b42ba0a2d05f391219c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:55.386643 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b82e65b31044b42ba0a2d05f391219c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:55.411243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b82e65b31044b42ba0a2d05f391219c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:55.430336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b82e65b31044b42ba0a2d05f391219c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:55.442975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1166/2329] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:10:55 2022] DELETE /identity/v3/users/0b82e65b31044b42ba0a2d05f391219c => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:55.450141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ab89c-09cf-4053-8af5-30deba16165a None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:55.450688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ab89c-09cf-4053-8af5-30deba16165a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:55.450933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ab89c-09cf-4053-8af5-30deba16165a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:55.484130 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-896ab89c-09cf-4053-8af5-30deba16165a None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:55.486240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-896ab89c-09cf-4053-8af5-30deba16165a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:55.566241 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1164/2330] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:55.577291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:55.578393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:55.654700 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b01705c-1405-4bd5-b613-0bbd4914630f admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:55.659996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e0c812-eb64-4ff6-a07d-9c93212cf073 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:55.661013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e0c812-eb64-4ff6-a07d-9c93212cf073 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:55.662164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e0c812-eb64-4ff6-a07d-9c93212cf073 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:55.662164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e0c812-eb64-4ff6-a07d-9c93212cf073 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:55.674263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e0c812-eb64-4ff6-a07d-9c93212cf073 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:55.676222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e0c812-eb64-4ff6-a07d-9c93212cf073 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:55.694881 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1167/2331] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:10:56.031499 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896ab89c-09cf-4053-8af5-30deba16165a None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:56.033922 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-896ab89c-09cf-4053-8af5-30deba16165a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:56.052047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896ab89c-09cf-4053-8af5-30deba16165a None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:56.058574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269370a9-ae82-4c35-882a-b8806c677707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:56.059595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269370a9-ae82-4c35-882a-b8806c677707 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:56.060301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269370a9-ae82-4c35-882a-b8806c677707 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:56.060777 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269370a9-ae82-4c35-882a-b8806c677707 admin admin] PATH_INFO: `/v3/projects/cacc8e9a98d94c72be4d3f28942e5ba3` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:56.062359 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269370a9-ae82-4c35-882a-b8806c677707 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cacc8e9a98d94c72be4d3f28942e5ba3)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:56.065042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269370a9-ae82-4c35-882a-b8806c677707 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:56.124423 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-269370a9-ae82-4c35-882a-b8806c677707 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cacc8e9a98d94c72be4d3f28942e5ba3', 'request_id': 'req-269370a9-ae82-4c35-882a-b8806c677707'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:56.133235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-269370a9-ae82-4c35-882a-b8806c677707 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cacc8e9a98d94c72be4d3f28942e5ba3', 'request_id': 'req-269370a9-ae82-4c35-882a-b8806c677707'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:56.144334 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-269370a9-ae82-4c35-882a-b8806c677707 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cacc8e9a98d94c72be4d3f28942e5ba3', 'request_id': 'req-269370a9-ae82-4c35-882a-b8806c677707'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:56.166872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-269370a9-ae82-4c35-882a-b8806c677707 admin admin] The token cache is being invalidate because project cacc8e9a98d94c72be4d3f28942e5ba3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:10:56.167392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-269370a9-ae82-4c35-882a-b8806c677707 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:56.168969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-269370a9-ae82-4c35-882a-b8806c677707 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:10:56.171106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1165/2332] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:10:56 2022] DELETE /identity/v3/projects/cacc8e9a98d94c72be4d3f28942e5ba3 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:10:56.178842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34574d9d-2513-43ac-8475-b8b9780d31ed None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:56.179552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34574d9d-2513-43ac-8475-b8b9780d31ed None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:56.179863 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34574d9d-2513-43ac-8475-b8b9780d31ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:56.216624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-34574d9d-2513-43ac-8475-b8b9780d31ed None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:56.218101 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-34574d9d-2513-43ac-8475-b8b9780d31ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:56.250026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:56.342226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1168/2333] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:56.350110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b44b3c-7bb8-4683-b6ef-61a1d6b9a96b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:56.350392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b44b3c-7bb8-4683-b6ef-61a1d6b9a96b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:56.350646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b44b3c-7bb8-4683-b6ef-61a1d6b9a96b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:56.375332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-49b44b3c-7bb8-4683-b6ef-61a1d6b9a96b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:56.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-49b44b3c-7bb8-4683-b6ef-61a1d6b9a96b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:56.388514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1166/2334] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:10:56.392834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34574d9d-2513-43ac-8475-b8b9780d31ed None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:56.405872 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34574d9d-2513-43ac-8475-b8b9780d31ed None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:56.411323 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85e8285-edb7-4ed2-a1f9-1ed14b7b4ea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:56.412008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85e8285-edb7-4ed2-a1f9-1ed14b7b4ea7 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:56.412414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85e8285-edb7-4ed2-a1f9-1ed14b7b4ea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:56.412594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85e8285-edb7-4ed2-a1f9-1ed14b7b4ea7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:56.413469 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85e8285-edb7-4ed2-a1f9-1ed14b7b4ea7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:10:56.413840 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85e8285-edb7-4ed2-a1f9-1ed14b7b4ea7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:56.415494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85e8285-edb7-4ed2-a1f9-1ed14b7b4ea7 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:56.430311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1169/2335] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:10:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:10:56.436393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b44b3c-7bb8-4683-b6ef-61a1d6b9a96b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:56.457370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b44b3c-7bb8-4683-b6ef-61a1d6b9a96b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:56.467111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18146651-78b5-403e-8fee-07ff13c70fe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:56.468070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18146651-78b5-403e-8fee-07ff13c70fe6 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:56.468958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18146651-78b5-403e-8fee-07ff13c70fe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:56.470294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18146651-78b5-403e-8fee-07ff13c70fe6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:56.470294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18146651-78b5-403e-8fee-07ff13c70fe6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:56.474981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18146651-78b5-403e-8fee-07ff13c70fe6 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:56.545247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1167/2336] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:10:56 2022] POST /identity/v3/projects => generated 383 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:10:56.555339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85e8285-edb7-4ed2-a1f9-1ed14b7b4ea7 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:56.570713 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85e8285-edb7-4ed2-a1f9-1ed14b7b4ea7 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:56.695775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63af9869-9f56-426d-945a-a067d39f9e9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:56.696852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63af9869-9f56-426d-945a-a067d39f9e9d admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:56.697564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63af9869-9f56-426d-945a-a067d39f9e9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:56.698135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63af9869-9f56-426d-945a-a067d39f9e9d admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:56.700482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63af9869-9f56-426d-945a-a067d39f9e9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:56.704695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63af9869-9f56-426d-945a-a067d39f9e9d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:56.737628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1170/2337] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:10:56 2022] POST /identity/v3/users => generated 292 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:10:56.742193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18146651-78b5-403e-8fee-07ff13c70fe6 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:56.753940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18146651-78b5-403e-8fee-07ff13c70fe6 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:56.765038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1300040a-ed36-4126-a14d-8b72f40d910c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:56.766346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1300040a-ed36-4126-a14d-8b72f40d910c admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:56.767171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1300040a-ed36-4126-a14d-8b72f40d910c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:56.767925 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1300040a-ed36-4126-a14d-8b72f40d910c admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:56.769260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1300040a-ed36-4126-a14d-8b72f40d910c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:56.771761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1300040a-ed36-4126-a14d-8b72f40d910c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:56.794638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1168/2338] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:56 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:10:56.794815 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63af9869-9f56-426d-945a-a067d39f9e9d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:56.803871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63af9869-9f56-426d-945a-a067d39f9e9d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:56.809482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc631e94-70e5-4bb4-a092-889fad32ffde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:56.810281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc631e94-70e5-4bb4-a092-889fad32ffde admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:56.810572 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc631e94-70e5-4bb4-a092-889fad32ffde admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:56.810846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc631e94-70e5-4bb4-a092-889fad32ffde admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:56.811599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc631e94-70e5-4bb4-a092-889fad32ffde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:56.813241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc631e94-70e5-4bb4-a092-889fad32ffde admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:56.825410 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1171/2339] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:10:56 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:10:56.830456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1300040a-ed36-4126-a14d-8b72f40d910c admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:56.842252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1300040a-ed36-4126-a14d-8b72f40d910c admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:56.847602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a906bd-24dc-4ac7-b016-d8711a3eb88c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:56.848458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a906bd-24dc-4ac7-b016-d8711a3eb88c admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:56.848796 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a906bd-24dc-4ac7-b016-d8711a3eb88c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:56.849134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a906bd-24dc-4ac7-b016-d8711a3eb88c admin admin] PATH_INFO: `/v3/projects/462dde0544874f1f939040196c594cd0/users/89ccdfd253914d5694418fab772897e8/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:56.872193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a906bd-24dc-4ac7-b016-d8711a3eb88c admin admin] RBAC: Authorizing `identity:create_grant(project_id=462dde0544874f1f939040196c594cd0, user_id=89ccdfd253914d5694418fab772897e8, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:56.875237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a906bd-24dc-4ac7-b016-d8711a3eb88c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:56.893766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1169/2340] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:10:56 2022] PUT /identity/v3/projects/462dde0544874f1f939040196c594cd0/users/89ccdfd253914d5694418fab772897e8/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:10:56.907435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fb434d-f270-4956-96e5-81dd9e1b86b5 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:56.907956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fb434d-f270-4956-96e5-81dd9e1b86b5 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:56.908518 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fb434d-f270-4956-96e5-81dd9e1b86b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:56.962959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-60fb434d-f270-4956-96e5-81dd9e1b86b5 None None] MFA Rules not processed for user `89ccdfd253914d5694418fab772897e8`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:10:56.965208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-60fb434d-f270-4956-96e5-81dd9e1b86b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:57.081959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1172/2341] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:10:56 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:10:57.095998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a906bd-24dc-4ac7-b016-d8711a3eb88c admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:10:57.098058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a7a906bd-24dc-4ac7-b016-d8711a3eb88c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:10:57.123680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:10:57.172049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a906bd-24dc-4ac7-b016-d8711a3eb88c admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:10:57.176238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d564803-e4d6-4297-988d-97f02fa6a21c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:10:57.176887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d564803-e4d6-4297-988d-97f02fa6a21c service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:10:57.177168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d564803-e4d6-4297-988d-97f02fa6a21c service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:10:57.177443 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d564803-e4d6-4297-988d-97f02fa6a21c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:10:57.192961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d564803-e4d6-4297-988d-97f02fa6a21c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:10:57.196091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d564803-e4d6-4297-988d-97f02fa6a21c service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:10:57.214905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1170/2342] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:10:57 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:11:00.094789 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fb434d-f270-4956-96e5-81dd9e1b86b5 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:00.118128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:11:00.197526 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fb434d-f270-4956-96e5-81dd9e1b86b5 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:00.202973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:00.203748 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:00.204066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:00.204375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] PATH_INFO: `/v3/users/89ccdfd253914d5694418fab772897e8` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:00.205974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89ccdfd253914d5694418fab772897e8)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:00.209115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:00.272604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89ccdfd253914d5694418fab772897e8'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:00.274009 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89ccdfd253914d5694418fab772897e8'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:00.284301 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89ccdfd253914d5694418fab772897e8'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:00.299900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89ccdfd253914d5694418fab772897e8'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:00.308163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89ccdfd253914d5694418fab772897e8'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:00.317743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1173/2343] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:11:00 2022] DELETE /identity/v3/users/89ccdfd253914d5694418fab772897e8 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:11:00.322481 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e908db98-30dd-415b-897c-af2e1632cd3f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:00.323003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e908db98-30dd-415b-897c-af2e1632cd3f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:00.323100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e908db98-30dd-415b-897c-af2e1632cd3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:00.351085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-e908db98-30dd-415b-897c-af2e1632cd3f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:00.352643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e908db98-30dd-415b-897c-af2e1632cd3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:00.471495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1171/2344] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:11:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:00.490178 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:00.492502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:00.636376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e0d0ce-15da-434a-8103-8f5fca342870 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:00.641079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69726f9-0104-45ff-b585-7dd79d469e42 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:00.641956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69726f9-0104-45ff-b585-7dd79d469e42 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:00.642262 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69726f9-0104-45ff-b585-7dd79d469e42 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:00.642614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69726f9-0104-45ff-b585-7dd79d469e42 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:00.656500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69726f9-0104-45ff-b585-7dd79d469e42 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:00.658055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69726f9-0104-45ff-b585-7dd79d469e42 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:00.670144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1174/2345] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:11:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:11:01.201652 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e908db98-30dd-415b-897c-af2e1632cd3f None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:01.203914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e908db98-30dd-415b-897c-af2e1632cd3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:01.234231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e908db98-30dd-415b-897c-af2e1632cd3f None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:01.240862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:01.241504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:01.241779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:01.242077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc admin admin] PATH_INFO: `/v3/projects/462dde0544874f1f939040196c594cd0` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:01.243289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc admin admin] RBAC: Authorizing `identity:delete_project(project_id=462dde0544874f1f939040196c594cd0)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:01.245299 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:01.298688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '462dde0544874f1f939040196c594cd0', 'request_id': 'req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:01.305957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '462dde0544874f1f939040196c594cd0', 'request_id': 'req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:01.313120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '462dde0544874f1f939040196c594cd0', 'request_id': 'req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:01.328972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc admin admin] The token cache is being invalidate because project 462dde0544874f1f939040196c594cd0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:11:01.329934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:01.330128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-63fffa3b-187a-4f4b-adbf-3a7df7c2bccc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:01.331661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1172/2346] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:11:01 2022] DELETE /identity/v3/projects/462dde0544874f1f939040196c594cd0 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:11:01.336957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a2110c-e4f1-4870-bb77-76882d261bb0 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:01.337541 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a2110c-e4f1-4870-bb77-76882d261bb0 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:01.337541 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a2110c-e4f1-4870-bb77-76882d261bb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:01.359308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a4a2110c-e4f1-4870-bb77-76882d261bb0 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:01.360564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a4a2110c-e4f1-4870-bb77-76882d261bb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:01.401645 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:11:01.434897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1175/2347] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:11:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:11:01.443538 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1dec60-4316-4c72-85f0-a7608927583b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:01.443824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1dec60-4316-4c72-85f0-a7608927583b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:01.444108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1dec60-4316-4c72-85f0-a7608927583b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:01.464531 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ea1dec60-4316-4c72-85f0-a7608927583b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:01.465785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ea1dec60-4316-4c72-85f0-a7608927583b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:01.473965 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1173/2348] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:11:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:01.478650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a2110c-e4f1-4870-bb77-76882d261bb0 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:01.489200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a2110c-e4f1-4870-bb77-76882d261bb0 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:01.493764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3778c860-b7ad-4836-a1b5-8014ab076bc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:01.494367 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3778c860-b7ad-4836-a1b5-8014ab076bc0 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:01.494651 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3778c860-b7ad-4836-a1b5-8014ab076bc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:01.494919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3778c860-b7ad-4836-a1b5-8014ab076bc0 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:01.495705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3778c860-b7ad-4836-a1b5-8014ab076bc0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:11:01.496703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3778c860-b7ad-4836-a1b5-8014ab076bc0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:01.498860 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3778c860-b7ad-4836-a1b5-8014ab076bc0 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:01.511557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1176/2349] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:11:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:11:01.516361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1dec60-4316-4c72-85f0-a7608927583b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:01.526859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1dec60-4316-4c72-85f0-a7608927583b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:01.534161 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53628ae-56b6-4627-b977-2c696a8bde04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:01.535234 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53628ae-56b6-4627-b977-2c696a8bde04 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:01.535772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53628ae-56b6-4627-b977-2c696a8bde04 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:01.536262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53628ae-56b6-4627-b977-2c696a8bde04 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:01.537487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53628ae-56b6-4627-b977-2c696a8bde04 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:01.541471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53628ae-56b6-4627-b977-2c696a8bde04 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:01.585655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1174/2350] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:11:01 2022] POST /identity/v3/projects => generated 371 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:11:01.589811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3778c860-b7ad-4836-a1b5-8014ab076bc0 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:01.600273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3778c860-b7ad-4836-a1b5-8014ab076bc0 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:01.712925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67a6987-29fe-4978-8663-08314b3cb8ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:01.715658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67a6987-29fe-4978-8663-08314b3cb8ec admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:01.716284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67a6987-29fe-4978-8663-08314b3cb8ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:01.716742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67a6987-29fe-4978-8663-08314b3cb8ec admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:01.717728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67a6987-29fe-4978-8663-08314b3cb8ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:01.720652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67a6987-29fe-4978-8663-08314b3cb8ec admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:01.762951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1177/2351] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:11:01 2022] POST /identity/v3/users => generated 286 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:11:01.766193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53628ae-56b6-4627-b977-2c696a8bde04 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:01.786402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53628ae-56b6-4627-b977-2c696a8bde04 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:01.795140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e1c361-49a4-4536-9c03-f83070ba0abf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:01.796288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e1c361-49a4-4536-9c03-f83070ba0abf admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:01.796804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e1c361-49a4-4536-9c03-f83070ba0abf admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:01.797291 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e1c361-49a4-4536-9c03-f83070ba0abf admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:01.798955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e1c361-49a4-4536-9c03-f83070ba0abf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:01.802172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e1c361-49a4-4536-9c03-f83070ba0abf admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:01.823280 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1175/2352] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:11:01 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:11:01.827726 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67a6987-29fe-4978-8663-08314b3cb8ec admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:01.840966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67a6987-29fe-4978-8663-08314b3cb8ec admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:01.847197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5ce205-a720-4692-9626-ab0f41e859a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:01.847197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5ce205-a720-4692-9626-ab0f41e859a8 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:01.847619 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5ce205-a720-4692-9626-ab0f41e859a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:01.852896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5ce205-a720-4692-9626-ab0f41e859a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:01.854260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5ce205-a720-4692-9626-ab0f41e859a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:01.858426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5ce205-a720-4692-9626-ab0f41e859a8 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:01.878241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1178/2353] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:11:01 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:11:01.879557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e1c361-49a4-4536-9c03-f83070ba0abf admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:01.891839 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e1c361-49a4-4536-9c03-f83070ba0abf admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:01.897707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14de7fcb-6c1a-4af8-a58e-11e8e3cf3b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:01.898514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14de7fcb-6c1a-4af8-a58e-11e8e3cf3b46 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:01.898853 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14de7fcb-6c1a-4af8-a58e-11e8e3cf3b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:01.899268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14de7fcb-6c1a-4af8-a58e-11e8e3cf3b46 admin admin] PATH_INFO: `/v3/projects/5efad2fec1c44d4a96661c9b4eb41610/users/7d3b7bb219f4463296d02f32258a6954/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:01.904127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:11:01.944433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14de7fcb-6c1a-4af8-a58e-11e8e3cf3b46 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5efad2fec1c44d4a96661c9b4eb41610, user_id=7d3b7bb219f4463296d02f32258a6954, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:01.947133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14de7fcb-6c1a-4af8-a58e-11e8e3cf3b46 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:01.963316 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1176/2354] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:11:01 2022] PUT /identity/v3/projects/5efad2fec1c44d4a96661c9b4eb41610/users/7d3b7bb219f4463296d02f32258a6954/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:11:01.975664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86ce3f6-eeec-4eee-8878-2ef2c965bc1b None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:01.975959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86ce3f6-eeec-4eee-8878-2ef2c965bc1b None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:01.976261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86ce3f6-eeec-4eee-8878-2ef2c965bc1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:02.018432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-d86ce3f6-eeec-4eee-8878-2ef2c965bc1b None None] MFA Rules not processed for user `7d3b7bb219f4463296d02f32258a6954`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:02.019950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d86ce3f6-eeec-4eee-8878-2ef2c965bc1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:02.133936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1179/2355] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:11:01 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:02.148356 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14de7fcb-6c1a-4af8-a58e-11e8e3cf3b46 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:02.151056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-14de7fcb-6c1a-4af8-a58e-11e8e3cf3b46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:02.286437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14de7fcb-6c1a-4af8-a58e-11e8e3cf3b46 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:02.293466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77af868-1f28-4469-8e71-af401e1ccf7d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4160039561434e51a87c6c7a019f2dff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:02.294780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77af868-1f28-4469-8e71-af401e1ccf7d service glance] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:02.295290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77af868-1f28-4469-8e71-af401e1ccf7d service glance] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:02.295838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77af868-1f28-4469-8e71-af401e1ccf7d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:02.311806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77af868-1f28-4469-8e71-af401e1ccf7d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:02.314253 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77af868-1f28-4469-8e71-af401e1ccf7d service glance] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:02.328018 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1177/2356] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:11:02 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:11:07.960863 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86ce3f6-eeec-4eee-8878-2ef2c965bc1b None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:07.968995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d86ce3f6-eeec-4eee-8878-2ef2c965bc1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:08.012826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:11:08.051106 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86ce3f6-eeec-4eee-8878-2ef2c965bc1b None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:08.057720 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f9da4b-6a71-4bbd-864a-097ddf017b90 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:08.058628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f9da4b-6a71-4bbd-864a-097ddf017b90 service placement] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:08.058905 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f9da4b-6a71-4bbd-864a-097ddf017b90 service placement] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:08.059182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f9da4b-6a71-4bbd-864a-097ddf017b90 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:08.061255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a7f9da4b-6a71-4bbd-864a-097ddf017b90 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:08.080816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f9da4b-6a71-4bbd-864a-097ddf017b90 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:08.082465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f9da4b-6a71-4bbd-864a-097ddf017b90 service placement] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:08.099722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1180/2357] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:11:07 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:11:09.907382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ca8d4-22ee-422a-adfd-44458903d8df None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:09.907967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ca8d4-22ee-422a-adfd-44458903d8df None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:09.907967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001ca8d4-22ee-422a-adfd-44458903d8df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:09.930560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-001ca8d4-22ee-422a-adfd-44458903d8df None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:09.931811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-001ca8d4-22ee-422a-adfd-44458903d8df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:09.997402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1178/2358] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:11:09 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:10.053117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f9da4b-6a71-4bbd-864a-097ddf017b90 service placement] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:10.054602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a7f9da4b-6a71-4bbd-864a-097ddf017b90 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:10.154797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f9da4b-6a71-4bbd-864a-097ddf017b90 service placement] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:10.156868 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a612e70-95a6-45ec-b585-59f5480f033f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:10.158160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a612e70-95a6-45ec-b585-59f5480f033f service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:10.158656 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a612e70-95a6-45ec-b585-59f5480f033f service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:10.159311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a612e70-95a6-45ec-b585-59f5480f033f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:10.172439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a612e70-95a6-45ec-b585-59f5480f033f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:10.174075 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a612e70-95a6-45ec-b585-59f5480f033f service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:10.184015 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1181/2359] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:11:10 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 01:11:11.012649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debfc622-b616-4935-ab54-752e80b4776a None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:11.013428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debfc622-b616-4935-ab54-752e80b4776a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:11.013428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debfc622-b616-4935-ab54-752e80b4776a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:11.032952 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-debfc622-b616-4935-ab54-752e80b4776a None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:11.034216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-debfc622-b616-4935-ab54-752e80b4776a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:11.041486 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1179/2360] 10.222.0.42 () {58 vars in 967 bytes} [Sun Jul 31 01:11:11 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:11.054768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a612e70-95a6-45ec-b585-59f5480f033f service swift] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:11.067999 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a612e70-95a6-45ec-b585-59f5480f033f service swift] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:11.073245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b897af12-5702-460e-9617-1b689aa9ad32 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:11.074477 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b897af12-5702-460e-9617-1b689aa9ad32 service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:11.074752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b897af12-5702-460e-9617-1b689aa9ad32 service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:11.075019 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b897af12-5702-460e-9617-1b689aa9ad32 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:11.084047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b897af12-5702-460e-9617-1b689aa9ad32 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:11.085607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b897af12-5702-460e-9617-1b689aa9ad32 service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:11.095065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1182/2361] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:11:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 01:11:11.410437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0586d688-ec77-49c0-b94f-24b5bd19022b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:11.410723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0586d688-ec77-49c0-b94f-24b5bd19022b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:11.411002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0586d688-ec77-49c0-b94f-24b5bd19022b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:11.437031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-0586d688-ec77-49c0-b94f-24b5bd19022b None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:11.438324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0586d688-ec77-49c0-b94f-24b5bd19022b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:11.445991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1180/2362] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:11:11 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:11.458314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b897af12-5702-460e-9617-1b689aa9ad32 service swift] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:11.469385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b897af12-5702-460e-9617-1b689aa9ad32 service swift] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:11.474652 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cba8d2-9ed6-4827-9a40-483c85760643 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:11.475281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cba8d2-9ed6-4827-9a40-483c85760643 service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:11.475596 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cba8d2-9ed6-4827-9a40-483c85760643 service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:11.475872 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cba8d2-9ed6-4827-9a40-483c85760643 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:11.485494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cba8d2-9ed6-4827-9a40-483c85760643 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:11.487111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cba8d2-9ed6-4827-9a40-483c85760643 service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:11.497289 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1183/2363] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:11:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 01:11:11.874123 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ddd257-bbe7-4c16-ba23-28e3083628c9 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:11.874439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ddd257-bbe7-4c16-ba23-28e3083628c9 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:11.874720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ddd257-bbe7-4c16-ba23-28e3083628c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:11.894704 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a0ddd257-bbe7-4c16-ba23-28e3083628c9 None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:11.895991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a0ddd257-bbe7-4c16-ba23-28e3083628c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:11.904145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1181/2364] 10.222.0.42 () {58 vars in 967 bytes} [Sun Jul 31 01:11:11 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:11.918200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cba8d2-9ed6-4827-9a40-483c85760643 service swift] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:11.937649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cba8d2-9ed6-4827-9a40-483c85760643 service swift] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:11.946767 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a4dad7-9ad8-4427-9e69-a2ebf7603a69 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:11.947878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a4dad7-9ad8-4427-9e69-a2ebf7603a69 service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:11.948385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a4dad7-9ad8-4427-9e69-a2ebf7603a69 service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:11.949372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a4dad7-9ad8-4427-9e69-a2ebf7603a69 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:11.963716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a4dad7-9ad8-4427-9e69-a2ebf7603a69 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:11.966017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a4dad7-9ad8-4427-9e69-a2ebf7603a69 service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:11.975481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1184/2365] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:11:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 01:11:19.165694 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bda7ff-85e4-4411-a8cb-b28d93bb9846 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:19.166755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bda7ff-85e4-4411-a8cb-b28d93bb9846 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:19.166755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bda7ff-85e4-4411-a8cb-b28d93bb9846 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:19.190048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-e0bda7ff-85e4-4411-a8cb-b28d93bb9846 None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:19.191235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e0bda7ff-85e4-4411-a8cb-b28d93bb9846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:19.199235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1182/2366] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:11:19 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:19.214905 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a4dad7-9ad8-4427-9e69-a2ebf7603a69 service swift] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:19.232437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a4dad7-9ad8-4427-9e69-a2ebf7603a69 service swift] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:19.238434 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415c8fd9-244d-4e84-84df-5f699f26dfe5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:19.239183 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415c8fd9-244d-4e84-84df-5f699f26dfe5 service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:19.239375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415c8fd9-244d-4e84-84df-5f699f26dfe5 service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:19.239660 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415c8fd9-244d-4e84-84df-5f699f26dfe5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:19.249200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415c8fd9-244d-4e84-84df-5f699f26dfe5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:19.250927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415c8fd9-244d-4e84-84df-5f699f26dfe5 service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:19.260735 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1185/2367] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:11:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 01:11:19.537847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bda7ff-85e4-4411-a8cb-b28d93bb9846 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:19.540104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e0bda7ff-85e4-4411-a8cb-b28d93bb9846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:19.710681 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bda7ff-85e4-4411-a8cb-b28d93bb9846 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:19.719571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:19.720717 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:19.721562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:19.721882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] PATH_INFO: `/v3/users/2e69ff835a0247869749c889143d8275` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:19.724353 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e69ff835a0247869749c889143d8275)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:19.728380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:19.738957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:11:19.875819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e69ff835a0247869749c889143d8275'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:19.898214 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e69ff835a0247869749c889143d8275'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:19.899819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e69ff835a0247869749c889143d8275'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:19.946096 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e69ff835a0247869749c889143d8275'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:19.947829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e69ff835a0247869749c889143d8275'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:20.007992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1183/2368] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:11:19 2022] DELETE /identity/v3/users/2e69ff835a0247869749c889143d8275 => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:11:20.014968 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d54ff49-1831-4d38-88f9-9e36420a9791 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:20.015302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d54ff49-1831-4d38-88f9-9e36420a9791 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:20.015944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d54ff49-1831-4d38-88f9-9e36420a9791 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:20.054956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-0d54ff49-1831-4d38-88f9-9e36420a9791 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:20.056682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0d54ff49-1831-4d38-88f9-9e36420a9791 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:20.191564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1186/2369] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:11:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:20.203306 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:20.205278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:20.384579 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e73708-5379-42ac-aa07-3ed61188baba admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:20.392949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10eb6e5-dac1-455c-bcf6-9e76b63ff3e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:20.394346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10eb6e5-dac1-455c-bcf6-9e76b63ff3e5 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:20.394864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10eb6e5-dac1-455c-bcf6-9e76b63ff3e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:20.395385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10eb6e5-dac1-455c-bcf6-9e76b63ff3e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:20.415770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10eb6e5-dac1-455c-bcf6-9e76b63ff3e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:20.418748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10eb6e5-dac1-455c-bcf6-9e76b63ff3e5 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:20.443450 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1184/2370] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:11:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:11:20.825461 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d54ff49-1831-4d38-88f9-9e36420a9791 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:20.827078 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0d54ff49-1831-4d38-88f9-9e36420a9791 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:20.836090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:11:20.843011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d54ff49-1831-4d38-88f9-9e36420a9791 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:20.859294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701341df-7272-462b-bebf-13da88c44745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:20.860898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701341df-7272-462b-bebf-13da88c44745 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:20.861757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701341df-7272-462b-bebf-13da88c44745 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:20.862271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701341df-7272-462b-bebf-13da88c44745 admin admin] PATH_INFO: `/v3/projects/bf5ca398012441cba969788634f537b8` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:20.864810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701341df-7272-462b-bebf-13da88c44745 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf5ca398012441cba969788634f537b8)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:20.868494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701341df-7272-462b-bebf-13da88c44745 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:21.070382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-701341df-7272-462b-bebf-13da88c44745 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf5ca398012441cba969788634f537b8', 'request_id': 'req-701341df-7272-462b-bebf-13da88c44745'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:21.083783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-701341df-7272-462b-bebf-13da88c44745 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf5ca398012441cba969788634f537b8', 'request_id': 'req-701341df-7272-462b-bebf-13da88c44745'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:21.098920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-701341df-7272-462b-bebf-13da88c44745 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf5ca398012441cba969788634f537b8', 'request_id': 'req-701341df-7272-462b-bebf-13da88c44745'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:21.119463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-701341df-7272-462b-bebf-13da88c44745 admin admin] The token cache is being invalidate because project bf5ca398012441cba969788634f537b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:11:21.119980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-701341df-7272-462b-bebf-13da88c44745 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:21.121155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-701341df-7272-462b-bebf-13da88c44745 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:21.127985 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1187/2371] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:11:20 2022] DELETE /identity/v3/projects/bf5ca398012441cba969788634f537b8 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:11:21.129554 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe997c0-810f-4f3c-a8b4-fa1e7a0914b2 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:21.129840 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe997c0-810f-4f3c-a8b4-fa1e7a0914b2 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:21.130114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe997c0-810f-4f3c-a8b4-fa1e7a0914b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:21.152471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-bfe997c0-810f-4f3c-a8b4-fa1e7a0914b2 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:21.153661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-bfe997c0-810f-4f3c-a8b4-fa1e7a0914b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:21.241914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1185/2372] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:11:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:11:21.254654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d691b3f3-4910-4b07-bbb4-3c2465eb8065 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:21.255022 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d691b3f3-4910-4b07-bbb4-3c2465eb8065 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:21.255484 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d691b3f3-4910-4b07-bbb4-3c2465eb8065 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:21.283095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-d691b3f3-4910-4b07-bbb4-3c2465eb8065 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:21.284459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d691b3f3-4910-4b07-bbb4-3c2465eb8065 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:21.296415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1188/2373] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:11:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:21.301910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe997c0-810f-4f3c-a8b4-fa1e7a0914b2 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:21.320326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe997c0-810f-4f3c-a8b4-fa1e7a0914b2 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:21.335060 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4fb145-2c90-4fc2-9b2d-dec87d66a331 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:21.336130 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4fb145-2c90-4fc2-9b2d-dec87d66a331 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:21.336699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4fb145-2c90-4fc2-9b2d-dec87d66a331 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:21.337357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4fb145-2c90-4fc2-9b2d-dec87d66a331 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:21.338272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4fb145-2c90-4fc2-9b2d-dec87d66a331 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:11:21.338647 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4fb145-2c90-4fc2-9b2d-dec87d66a331 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:21.340811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4fb145-2c90-4fc2-9b2d-dec87d66a331 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:21.362194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1186/2374] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:11:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:11:21.366833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d691b3f3-4910-4b07-bbb4-3c2465eb8065 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:21.377920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d691b3f3-4910-4b07-bbb4-3c2465eb8065 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:21.383643 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0c8f23-53d6-448e-a852-18813352b264 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:21.384429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0c8f23-53d6-448e-a852-18813352b264 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:21.384737 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0c8f23-53d6-448e-a852-18813352b264 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:21.385010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0c8f23-53d6-448e-a852-18813352b264 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:21.386340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0c8f23-53d6-448e-a852-18813352b264 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:21.388472 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0c8f23-53d6-448e-a852-18813352b264 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:21.424666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1189/2375] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:11:21 2022] POST /identity/v3/projects => generated 391 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:11:21.425921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4fb145-2c90-4fc2-9b2d-dec87d66a331 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:21.437660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4fb145-2c90-4fc2-9b2d-dec87d66a331 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:21.447734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:11:21.516716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4358d6-7ede-42e6-acc0-7151ff789051 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:21.517917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4358d6-7ede-42e6-acc0-7151ff789051 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:21.518570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4358d6-7ede-42e6-acc0-7151ff789051 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:21.519124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4358d6-7ede-42e6-acc0-7151ff789051 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:21.527203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4358d6-7ede-42e6-acc0-7151ff789051 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:21.530088 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4358d6-7ede-42e6-acc0-7151ff789051 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:21.567988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1187/2376] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:11:21 2022] POST /identity/v3/users => generated 296 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:11:21.568883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0c8f23-53d6-448e-a852-18813352b264 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:21.589578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0c8f23-53d6-448e-a852-18813352b264 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:21.598264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166d55f-27fa-483f-b7c7-3560a78b226c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:21.599847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166d55f-27fa-483f-b7c7-3560a78b226c admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:21.599847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166d55f-27fa-483f-b7c7-3560a78b226c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:21.599847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166d55f-27fa-483f-b7c7-3560a78b226c admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:21.601513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166d55f-27fa-483f-b7c7-3560a78b226c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:21.603701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166d55f-27fa-483f-b7c7-3560a78b226c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:21.629581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1190/2377] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:11:21 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:11:21.637103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4358d6-7ede-42e6-acc0-7151ff789051 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:21.649249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4358d6-7ede-42e6-acc0-7151ff789051 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:21.661445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f259f2-192a-42bf-bfe8-ece1111605ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:21.661445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f259f2-192a-42bf-bfe8-ece1111605ba admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:21.661445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f259f2-192a-42bf-bfe8-ece1111605ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:21.661445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f259f2-192a-42bf-bfe8-ece1111605ba admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:21.661954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f259f2-192a-42bf-bfe8-ece1111605ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:21.661954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f259f2-192a-42bf-bfe8-ece1111605ba admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:21.675550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1188/2378] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:11:21 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:11:21.684050 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166d55f-27fa-483f-b7c7-3560a78b226c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:21.707253 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166d55f-27fa-483f-b7c7-3560a78b226c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:21.724141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e168a5-c408-47b5-a336-6e79e67f5ebc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:21.724839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e168a5-c408-47b5-a336-6e79e67f5ebc admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:21.725116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e168a5-c408-47b5-a336-6e79e67f5ebc admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:21.725415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e168a5-c408-47b5-a336-6e79e67f5ebc admin admin] PATH_INFO: `/v3/projects/27149536afb94f62a84d33f6a65a54a3/users/9aff3344928f416ba7c35a9bda86caf2/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:21.757252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e168a5-c408-47b5-a336-6e79e67f5ebc admin admin] RBAC: Authorizing `identity:create_grant(project_id=27149536afb94f62a84d33f6a65a54a3, user_id=9aff3344928f416ba7c35a9bda86caf2, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:21.763848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e168a5-c408-47b5-a336-6e79e67f5ebc admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:21.805924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1191/2379] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:11:21 2022] PUT /identity/v3/projects/27149536afb94f62a84d33f6a65a54a3/users/9aff3344928f416ba7c35a9bda86caf2/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:11:21.885112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3650246a-b9ec-492b-a961-339a600f7939 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:21.885407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3650246a-b9ec-492b-a961-339a600f7939 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:21.887637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3650246a-b9ec-492b-a961-339a600f7939 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:21.967016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-3650246a-b9ec-492b-a961-339a600f7939 None None] MFA Rules not processed for user `9aff3344928f416ba7c35a9bda86caf2`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:21.969097 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-3650246a-b9ec-492b-a961-339a600f7939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:22.091129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1189/2380] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:11:21 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:22.098677 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e168a5-c408-47b5-a336-6e79e67f5ebc admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:22.106507 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-99e168a5-c408-47b5-a336-6e79e67f5ebc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:22.196996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e168a5-c408-47b5-a336-6e79e67f5ebc admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:22.201115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9701700c-2a1e-45d9-954f-d0c8b53b2f5a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:22.201749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9701700c-2a1e-45d9-954f-d0c8b53b2f5a service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:22.202028 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9701700c-2a1e-45d9-954f-d0c8b53b2f5a service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:22.202305 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9701700c-2a1e-45d9-954f-d0c8b53b2f5a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:22.218949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9701700c-2a1e-45d9-954f-d0c8b53b2f5a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:22.221066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9701700c-2a1e-45d9-954f-d0c8b53b2f5a service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:22.245317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1192/2381] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:11:22 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:11:32.083659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=60,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:11:32.235157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=79,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:11:38.472902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01e2d47-18a5-417d-b12c-fbc16352ffd6 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:38.472902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01e2d47-18a5-417d-b12c-fbc16352ffd6 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:38.486261 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01e2d47-18a5-417d-b12c-fbc16352ffd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:38.588561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b01e2d47-18a5-417d-b12c-fbc16352ffd6 None None] MFA Rules not processed for user `20df8d74c87143dd81dbaae76b3c2b62`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:38.591294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b01e2d47-18a5-417d-b12c-fbc16352ffd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:38.744032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1190/2382] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:11:38 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:41.282263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e201bd-3df0-44eb-b7d8-2dd738eeb780 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:41.282899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e201bd-3df0-44eb-b7d8-2dd738eeb780 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:41.282899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e201bd-3df0-44eb-b7d8-2dd738eeb780 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:41.305062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-c0e201bd-3df0-44eb-b7d8-2dd738eeb780 None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:41.306327 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c0e201bd-3df0-44eb-b7d8-2dd738eeb780 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:41.374046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1193/2383] 10.222.0.42 () {58 vars in 967 bytes} [Sun Jul 31 01:11:41 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:11:41.386926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01e2d47-18a5-417d-b12c-fbc16352ffd6 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:41.388425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b01e2d47-18a5-417d-b12c-fbc16352ffd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:41.472348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01e2d47-18a5-417d-b12c-fbc16352ffd6 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:41.477379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a61e2d-fdeb-429d-9af8-a4f8b4ba24fa service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:41.478019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a61e2d-fdeb-429d-9af8-a4f8b4ba24fa service swift] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:41.478299 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a61e2d-fdeb-429d-9af8-a4f8b4ba24fa service swift] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:41.479267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a61e2d-fdeb-429d-9af8-a4f8b4ba24fa service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:41.488531 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a61e2d-fdeb-429d-9af8-a4f8b4ba24fa service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:41.490177 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a61e2d-fdeb-429d-9af8-a4f8b4ba24fa service swift] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:41.504129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1191/2384] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:11:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 01:11:49.795268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e201bd-3df0-44eb-b7d8-2dd738eeb780 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:49.799224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c0e201bd-3df0-44eb-b7d8-2dd738eeb780 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:49.975574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e201bd-3df0-44eb-b7d8-2dd738eeb780 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:49.982044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9660c64e-bf57-4b88-b135-2ca9c3ef6419 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:49.982714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9660c64e-bf57-4b88-b135-2ca9c3ef6419 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:49.982995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9660c64e-bf57-4b88-b135-2ca9c3ef6419 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:49.983273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9660c64e-bf57-4b88-b135-2ca9c3ef6419 admin admin] PATH_INFO: `/v3/users/8fcc34c42bb64200ac5f40d770d6784b` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:49.984657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9660c64e-bf57-4b88-b135-2ca9c3ef6419 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8fcc34c42bb64200ac5f40d770d6784b)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:49.986930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9660c64e-bf57-4b88-b135-2ca9c3ef6419 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:50.091076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-9660c64e-bf57-4b88-b135-2ca9c3ef6419 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8fcc34c42bb64200ac5f40d770d6784b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:50.092375 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-9660c64e-bf57-4b88-b135-2ca9c3ef6419 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8fcc34c42bb64200ac5f40d770d6784b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:50.102539 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-9660c64e-bf57-4b88-b135-2ca9c3ef6419 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8fcc34c42bb64200ac5f40d770d6784b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:50.124217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-9660c64e-bf57-4b88-b135-2ca9c3ef6419 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8fcc34c42bb64200ac5f40d770d6784b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:50.135680 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-9660c64e-bf57-4b88-b135-2ca9c3ef6419 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8fcc34c42bb64200ac5f40d770d6784b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:50.143061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1194/2385] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:11:49 2022] DELETE /identity/v3/users/8fcc34c42bb64200ac5f40d770d6784b => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:11:50.406741 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a61e2d-fdeb-429d-9af8-a4f8b4ba24fa service swift] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:50.450728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a61e2d-fdeb-429d-9af8-a4f8b4ba24fa service swift] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:50.616417 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d235ca6-a480-446e-8f20-014cf5032668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:50.617254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d235ca6-a480-446e-8f20-014cf5032668 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:50.617810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d235ca6-a480-446e-8f20-014cf5032668 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:50.618548 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d235ca6-a480-446e-8f20-014cf5032668 admin admin] PATH_INFO: `/v3/projects/de83743a559b4a289aa40c8d16641ee9` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:50.621176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d235ca6-a480-446e-8f20-014cf5032668 admin admin] RBAC: Authorizing `identity:delete_project(project_id=de83743a559b4a289aa40c8d16641ee9)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:50.625389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d235ca6-a480-446e-8f20-014cf5032668 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:50.703246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:11:50.721354 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0d235ca6-a480-446e-8f20-014cf5032668 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de83743a559b4a289aa40c8d16641ee9', 'request_id': 'req-0d235ca6-a480-446e-8f20-014cf5032668'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:50.732907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0d235ca6-a480-446e-8f20-014cf5032668 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de83743a559b4a289aa40c8d16641ee9', 'request_id': 'req-0d235ca6-a480-446e-8f20-014cf5032668'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:50.746493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0d235ca6-a480-446e-8f20-014cf5032668 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de83743a559b4a289aa40c8d16641ee9', 'request_id': 'req-0d235ca6-a480-446e-8f20-014cf5032668'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:50.773520 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0d235ca6-a480-446e-8f20-014cf5032668 admin admin] The token cache is being invalidate because project de83743a559b4a289aa40c8d16641ee9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:11:50.773991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0d235ca6-a480-446e-8f20-014cf5032668 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:50.774986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-0d235ca6-a480-446e-8f20-014cf5032668 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:11:50.777018 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1192/2386] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:11:50 2022] DELETE /identity/v3/projects/de83743a559b4a289aa40c8d16641ee9 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:11:50.782606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b59248-d8fa-42d8-bcac-7a843e1b44df None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:50.782863 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b59248-d8fa-42d8-bcac-7a843e1b44df None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:50.783266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b59248-d8fa-42d8-bcac-7a843e1b44df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:50.817343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-73b59248-d8fa-42d8-bcac-7a843e1b44df None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:50.821096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-73b59248-d8fa-42d8-bcac-7a843e1b44df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:50.864128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:11:50.966070 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1195/2387] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:11:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:50.984322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e931154f-7aea-4033-b2fa-37c9f3445cee None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:50.984791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e931154f-7aea-4033-b2fa-37c9f3445cee None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:50.985268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e931154f-7aea-4033-b2fa-37c9f3445cee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:51.042714 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-e931154f-7aea-4033-b2fa-37c9f3445cee None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:51.044872 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e931154f-7aea-4033-b2fa-37c9f3445cee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:51.069071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1193/2388] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:11:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:51.071454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b59248-d8fa-42d8-bcac-7a843e1b44df None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:51.099585 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b59248-d8fa-42d8-bcac-7a843e1b44df None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:51.109910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf004d-0a0d-4f1d-8704-284e3ec50149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:51.111671 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf004d-0a0d-4f1d-8704-284e3ec50149 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:51.112367 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf004d-0a0d-4f1d-8704-284e3ec50149 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:51.113083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf004d-0a0d-4f1d-8704-284e3ec50149 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:51.114833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf004d-0a0d-4f1d-8704-284e3ec50149 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:11:51.115635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf004d-0a0d-4f1d-8704-284e3ec50149 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:51.119382 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf004d-0a0d-4f1d-8704-284e3ec50149 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:51.146139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1196/2389] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:11:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:11:51.154752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e931154f-7aea-4033-b2fa-37c9f3445cee None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:51.163995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e931154f-7aea-4033-b2fa-37c9f3445cee None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:51.170410 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f90fd2-cd94-4c15-9f72-248ff02b322d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:51.171059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f90fd2-cd94-4c15-9f72-248ff02b322d admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:51.171484 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f90fd2-cd94-4c15-9f72-248ff02b322d admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:51.171932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f90fd2-cd94-4c15-9f72-248ff02b322d admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:51.172907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f90fd2-cd94-4c15-9f72-248ff02b322d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:51.176459 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f90fd2-cd94-4c15-9f72-248ff02b322d admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:51.223255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf004d-0a0d-4f1d-8704-284e3ec50149 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:51.223754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1194/2390] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:11:51 2022] POST /identity/v3/projects => generated 369 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:11:51.240313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf004d-0a0d-4f1d-8704-284e3ec50149 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:51.334502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4741684f-9b6b-48ce-aeac-a5fed7f62eea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:51.334502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4741684f-9b6b-48ce-aeac-a5fed7f62eea admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:51.334502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4741684f-9b6b-48ce-aeac-a5fed7f62eea admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:51.334502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4741684f-9b6b-48ce-aeac-a5fed7f62eea admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:51.335258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4741684f-9b6b-48ce-aeac-a5fed7f62eea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:51.335258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4741684f-9b6b-48ce-aeac-a5fed7f62eea admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:51.367890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1197/2391] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:11:51 2022] POST /identity/v3/users => generated 285 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:11:51.375418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f90fd2-cd94-4c15-9f72-248ff02b322d admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:51.385167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f90fd2-cd94-4c15-9f72-248ff02b322d admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:51.390957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf4de89-6566-49b3-9248-a00d5620fce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:51.391773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf4de89-6566-49b3-9248-a00d5620fce7 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:51.392128 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf4de89-6566-49b3-9248-a00d5620fce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:51.392507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf4de89-6566-49b3-9248-a00d5620fce7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:51.393321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf4de89-6566-49b3-9248-a00d5620fce7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:51.401965 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf4de89-6566-49b3-9248-a00d5620fce7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:51.420794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1195/2392] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:11:51.425523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4741684f-9b6b-48ce-aeac-a5fed7f62eea admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:51.443012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4741684f-9b6b-48ce-aeac-a5fed7f62eea admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:51.451967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bfe624-3dd7-48c2-ab7c-29a18328b4ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:51.453188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bfe624-3dd7-48c2-ab7c-29a18328b4ed admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:51.453777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bfe624-3dd7-48c2-ab7c-29a18328b4ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:51.454357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bfe624-3dd7-48c2-ab7c-29a18328b4ed admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:51.455826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02bfe624-3dd7-48c2-ab7c-29a18328b4ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:51.458948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02bfe624-3dd7-48c2-ab7c-29a18328b4ed admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:51.482410 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1198/2393] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:11:51.489108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf4de89-6566-49b3-9248-a00d5620fce7 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:51.502483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf4de89-6566-49b3-9248-a00d5620fce7 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:51.508565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14242330-e6cc-4c0d-8143-b1c22da3b23d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:51.510130 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14242330-e6cc-4c0d-8143-b1c22da3b23d admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:51.510579 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14242330-e6cc-4c0d-8143-b1c22da3b23d admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:51.511031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14242330-e6cc-4c0d-8143-b1c22da3b23d admin admin] PATH_INFO: `/v3/projects/d25c5c4a187645eaa71c6f2015af6b57/users/b4d5926ba9644db5bda5c7feeaf5d60c/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:51.534605 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14242330-e6cc-4c0d-8143-b1c22da3b23d admin admin] RBAC: Authorizing `identity:create_grant(project_id=d25c5c4a187645eaa71c6f2015af6b57, user_id=b4d5926ba9644db5bda5c7feeaf5d60c, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:51.540237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14242330-e6cc-4c0d-8143-b1c22da3b23d admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:51.566023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1196/2394] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:11:51 2022] PUT /identity/v3/projects/d25c5c4a187645eaa71c6f2015af6b57/users/b4d5926ba9644db5bda5c7feeaf5d60c/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:11:51.646880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc569d9-af39-4447-a170-6ac00f1247be None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:51.647157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc569d9-af39-4447-a170-6ac00f1247be None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:51.647468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc569d9-af39-4447-a170-6ac00f1247be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:51.680188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-9dc569d9-af39-4447-a170-6ac00f1247be None None] MFA Rules not processed for user `b4d5926ba9644db5bda5c7feeaf5d60c`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:11:51.681490 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9dc569d9-af39-4447-a170-6ac00f1247be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:51.765595 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1199/2395] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:11:51 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:11:51.778332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14242330-e6cc-4c0d-8143-b1c22da3b23d admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:11:51.780445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-14242330-e6cc-4c0d-8143-b1c22da3b23d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:11:51.851492 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14242330-e6cc-4c0d-8143-b1c22da3b23d admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:11:51.856899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b004f5c5-e755-4289-ae49-7b7a80e06d59 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4160039561434e51a87c6c7a019f2dff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:11:51.857561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b004f5c5-e755-4289-ae49-7b7a80e06d59 service glance] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:11:51.857836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b004f5c5-e755-4289-ae49-7b7a80e06d59 service glance] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:11:51.858105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b004f5c5-e755-4289-ae49-7b7a80e06d59 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:11:51.867742 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b004f5c5-e755-4289-ae49-7b7a80e06d59 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:11:51.869649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b004f5c5-e755-4289-ae49-7b7a80e06d59 service glance] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:11:51.880871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1197/2396] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:11:51 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:12:00.210226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc569d9-af39-4447-a170-6ac00f1247be None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:00.214805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9dc569d9-af39-4447-a170-6ac00f1247be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:00.285734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:00.415730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc569d9-af39-4447-a170-6ac00f1247be None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:00.425172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:00.426282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:00.426772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:00.427261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] PATH_INFO: `/v3/users/9aff3344928f416ba7c35a9bda86caf2` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:00.429776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=9aff3344928f416ba7c35a9bda86caf2)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:00.433994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:00.549194 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9aff3344928f416ba7c35a9bda86caf2'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:00.550352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9aff3344928f416ba7c35a9bda86caf2'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:00.566712 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9aff3344928f416ba7c35a9bda86caf2'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:00.599801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9aff3344928f416ba7c35a9bda86caf2'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:00.616647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9aff3344928f416ba7c35a9bda86caf2'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:00.629927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1200/2397] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:12:00 2022] DELETE /identity/v3/users/9aff3344928f416ba7c35a9bda86caf2 => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:12:00.636707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74292a4a-b394-4b35-9814-35b6db2b955c None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:00.637431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74292a4a-b394-4b35-9814-35b6db2b955c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:00.637431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74292a4a-b394-4b35-9814-35b6db2b955c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:00.664553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-74292a4a-b394-4b35-9814-35b6db2b955c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:00.665950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-74292a4a-b394-4b35-9814-35b6db2b955c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:00.763008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1198/2398] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:00.776700 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:00.779188 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:00.883835 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18292269-4688-438b-8cb9-640c75880f3d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:00.890161 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f349d8d-dbe9-4d92-bad9-c60c254f0714 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:00.891181 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f349d8d-dbe9-4d92-bad9-c60c254f0714 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:00.891610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f349d8d-dbe9-4d92-bad9-c60c254f0714 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:00.891999 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f349d8d-dbe9-4d92-bad9-c60c254f0714 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:00.907721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f349d8d-dbe9-4d92-bad9-c60c254f0714 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:00.910846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f349d8d-dbe9-4d92-bad9-c60c254f0714 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:00.938064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1201/2399] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:12:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:12:01.358243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74292a4a-b394-4b35-9814-35b6db2b955c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:01.360892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-74292a4a-b394-4b35-9814-35b6db2b955c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:01.379903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:01.389480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74292a4a-b394-4b35-9814-35b6db2b955c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:01.425237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667e2ad3-ab86-4444-91bb-1d71163aa4dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:01.427062 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667e2ad3-ab86-4444-91bb-1d71163aa4dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:01.428785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667e2ad3-ab86-4444-91bb-1d71163aa4dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:01.431490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667e2ad3-ab86-4444-91bb-1d71163aa4dc admin admin] PATH_INFO: `/v3/projects/27149536afb94f62a84d33f6a65a54a3` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:01.433696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667e2ad3-ab86-4444-91bb-1d71163aa4dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=27149536afb94f62a84d33f6a65a54a3)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:01.435986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667e2ad3-ab86-4444-91bb-1d71163aa4dc admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:01.489267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-667e2ad3-ab86-4444-91bb-1d71163aa4dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27149536afb94f62a84d33f6a65a54a3', 'request_id': 'req-667e2ad3-ab86-4444-91bb-1d71163aa4dc'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:01.496579 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-667e2ad3-ab86-4444-91bb-1d71163aa4dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27149536afb94f62a84d33f6a65a54a3', 'request_id': 'req-667e2ad3-ab86-4444-91bb-1d71163aa4dc'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:01.502629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-667e2ad3-ab86-4444-91bb-1d71163aa4dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27149536afb94f62a84d33f6a65a54a3', 'request_id': 'req-667e2ad3-ab86-4444-91bb-1d71163aa4dc'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:01.517982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-667e2ad3-ab86-4444-91bb-1d71163aa4dc admin admin] The token cache is being invalidate because project 27149536afb94f62a84d33f6a65a54a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:12:01.518540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-667e2ad3-ab86-4444-91bb-1d71163aa4dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:01.519306 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-667e2ad3-ab86-4444-91bb-1d71163aa4dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:01.523963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1199/2400] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:12:01 2022] DELETE /identity/v3/projects/27149536afb94f62a84d33f6a65a54a3 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:12:01.531963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19829b90-e341-42ef-a043-050a2d23a68f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:01.532537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19829b90-e341-42ef-a043-050a2d23a68f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:01.532537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19829b90-e341-42ef-a043-050a2d23a68f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:01.554225 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-19829b90-e341-42ef-a043-050a2d23a68f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:01.557241 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-19829b90-e341-42ef-a043-050a2d23a68f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:01.599785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:01.639431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1202/2401] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:01.641223 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1725ca5-663a-4bed-8ce2-c256120057df None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:01.641963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1725ca5-663a-4bed-8ce2-c256120057df None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:01.641963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1725ca5-663a-4bed-8ce2-c256120057df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:01.679438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d1725ca5-663a-4bed-8ce2-c256120057df None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:01.690296 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d1725ca5-663a-4bed-8ce2-c256120057df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:01.700754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1200/2402] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:12:01.705486 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19829b90-e341-42ef-a043-050a2d23a68f None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:01.717746 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19829b90-e341-42ef-a043-050a2d23a68f None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:01.723386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0a69d6-8ad6-4d84-a3b8-08434dfa6b66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:01.724059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0a69d6-8ad6-4d84-a3b8-08434dfa6b66 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:01.724343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0a69d6-8ad6-4d84-a3b8-08434dfa6b66 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:01.724624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0a69d6-8ad6-4d84-a3b8-08434dfa6b66 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:01.725339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0a69d6-8ad6-4d84-a3b8-08434dfa6b66 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:12:01.725646 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0a69d6-8ad6-4d84-a3b8-08434dfa6b66 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:01.727288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0a69d6-8ad6-4d84-a3b8-08434dfa6b66 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:01.742245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1203/2403] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:12:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:12:01.748545 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1725ca5-663a-4bed-8ce2-c256120057df None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:01.761905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1725ca5-663a-4bed-8ce2-c256120057df None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:01.769265 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8aa9d8d-ce66-467a-87b4-dd6d455ceb27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:01.769971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8aa9d8d-ce66-467a-87b4-dd6d455ceb27 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:01.770330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8aa9d8d-ce66-467a-87b4-dd6d455ceb27 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:01.771035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8aa9d8d-ce66-467a-87b4-dd6d455ceb27 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:01.771539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8aa9d8d-ce66-467a-87b4-dd6d455ceb27 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:01.773789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8aa9d8d-ce66-467a-87b4-dd6d455ceb27 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:01.829498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1201/2404] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:12:01 2022] POST /identity/v3/projects => generated 383 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:12:01.834158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0a69d6-8ad6-4d84-a3b8-08434dfa6b66 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:01.853964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0a69d6-8ad6-4d84-a3b8-08434dfa6b66 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:01.936703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813714c5-b340-4e6f-86c6-37fe90164d20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:01.937346 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813714c5-b340-4e6f-86c6-37fe90164d20 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:01.937626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813714c5-b340-4e6f-86c6-37fe90164d20 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:01.938118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813714c5-b340-4e6f-86c6-37fe90164d20 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:01.938852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813714c5-b340-4e6f-86c6-37fe90164d20 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:01.941072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813714c5-b340-4e6f-86c6-37fe90164d20 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:01.981686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1204/2405] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:12:01 2022] POST /identity/v3/users => generated 292 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:12:01.983849 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8aa9d8d-ce66-467a-87b4-dd6d455ceb27 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:01.996425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8aa9d8d-ce66-467a-87b4-dd6d455ceb27 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:02.001867 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82d4d1f-54d9-455a-8a82-de2c3be2e9fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:02.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82d4d1f-54d9-455a-8a82-de2c3be2e9fd admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:02.003019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82d4d1f-54d9-455a-8a82-de2c3be2e9fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:02.003497 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82d4d1f-54d9-455a-8a82-de2c3be2e9fd admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:02.004322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82d4d1f-54d9-455a-8a82-de2c3be2e9fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:02.006919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82d4d1f-54d9-455a-8a82-de2c3be2e9fd admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:02.033065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1202/2406] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:12:01 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:12:02.034208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813714c5-b340-4e6f-86c6-37fe90164d20 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:02.046606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813714c5-b340-4e6f-86c6-37fe90164d20 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:02.052407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36f8601-2c51-4dde-b11b-0c0f662d77d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:02.053417 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36f8601-2c51-4dde-b11b-0c0f662d77d2 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:02.053892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36f8601-2c51-4dde-b11b-0c0f662d77d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:02.054258 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36f8601-2c51-4dde-b11b-0c0f662d77d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:02.055196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36f8601-2c51-4dde-b11b-0c0f662d77d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:02.057672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36f8601-2c51-4dde-b11b-0c0f662d77d2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:02.071897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1205/2407] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:12:02 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:12:02.076435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82d4d1f-54d9-455a-8a82-de2c3be2e9fd admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:02.088062 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82d4d1f-54d9-455a-8a82-de2c3be2e9fd admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:02.093259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835223a5-416c-499b-85d5-9a37d597418e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:02.093947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835223a5-416c-499b-85d5-9a37d597418e admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:02.094217 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835223a5-416c-499b-85d5-9a37d597418e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:02.094485 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835223a5-416c-499b-85d5-9a37d597418e admin admin] PATH_INFO: `/v3/projects/b018ba41816049feaa75e10489ec5281/users/1776c123f98c44b69807ec2b0281b718/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:02.127026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835223a5-416c-499b-85d5-9a37d597418e admin admin] RBAC: Authorizing `identity:create_grant(project_id=b018ba41816049feaa75e10489ec5281, user_id=1776c123f98c44b69807ec2b0281b718, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:02.132550 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835223a5-416c-499b-85d5-9a37d597418e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:02.151975 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1203/2408] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:12:02 2022] PUT /identity/v3/projects/b018ba41816049feaa75e10489ec5281/users/1776c123f98c44b69807ec2b0281b718/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:12:02.157787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccea590e-aec6-499e-bdda-12b29ae82a9c None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:02.158278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccea590e-aec6-499e-bdda-12b29ae82a9c None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:02.158744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccea590e-aec6-499e-bdda-12b29ae82a9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:02.192360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ccea590e-aec6-499e-bdda-12b29ae82a9c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:02.193602 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ccea590e-aec6-499e-bdda-12b29ae82a9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:02.271582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1206/2409] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:02.284870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835223a5-416c-499b-85d5-9a37d597418e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:02.287696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-835223a5-416c-499b-85d5-9a37d597418e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:02.364444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835223a5-416c-499b-85d5-9a37d597418e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:02.368045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1924412c-bcbb-4d0e-885d-76bceddaee43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:02.369215 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1924412c-bcbb-4d0e-885d-76bceddaee43 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:02.369483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1924412c-bcbb-4d0e-885d-76bceddaee43 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:02.369755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1924412c-bcbb-4d0e-885d-76bceddaee43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:02.385947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1924412c-bcbb-4d0e-885d-76bceddaee43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:02.387411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1924412c-bcbb-4d0e-885d-76bceddaee43 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:02.397747 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1204/2410] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:12:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:12:03.730171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccea590e-aec6-499e-bdda-12b29ae82a9c None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:03.734318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ccea590e-aec6-499e-bdda-12b29ae82a9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:03.760707 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccea590e-aec6-499e-bdda-12b29ae82a9c None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:03.772524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:03.775220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:03.776075 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:03.776822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] PATH_INFO: `/v3/users/7d3b7bb219f4463296d02f32258a6954` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:03.780159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] RBAC: Authorizing `identity:delete_user(user_id=7d3b7bb219f4463296d02f32258a6954)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:03.785683 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:03.919714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:03.927794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7d3b7bb219f4463296d02f32258a6954'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:03.930505 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7d3b7bb219f4463296d02f32258a6954'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:03.959403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7d3b7bb219f4463296d02f32258a6954'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:03.986357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7d3b7bb219f4463296d02f32258a6954'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:04.010994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7d3b7bb219f4463296d02f32258a6954'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:04.024229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1207/2411] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:12:03 2022] DELETE /identity/v3/users/7d3b7bb219f4463296d02f32258a6954 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:12:04.031568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a697489-67d9-42dd-ac8b-74a0f45de889 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:04.032454 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a697489-67d9-42dd-ac8b-74a0f45de889 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:04.032454 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a697489-67d9-42dd-ac8b-74a0f45de889 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:04.053411 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-7a697489-67d9-42dd-ac8b-74a0f45de889 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:04.054672 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7a697489-67d9-42dd-ac8b-74a0f45de889 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:04.086604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:04.159434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1205/2412] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:04.184910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:04.187458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:04.318137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30433ead-b30d-479f-ab27-4cdb57c91fca admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:04.327982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7414b1-87fa-4273-9cb4-d69da3899ef2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:04.328625 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7414b1-87fa-4273-9cb4-d69da3899ef2 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:04.328931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7414b1-87fa-4273-9cb4-d69da3899ef2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:04.329270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7414b1-87fa-4273-9cb4-d69da3899ef2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:04.358815 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7414b1-87fa-4273-9cb4-d69da3899ef2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:04.360242 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7414b1-87fa-4273-9cb4-d69da3899ef2 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:04.382047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1208/2413] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:12:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:12:05.610148 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a697489-67d9-42dd-ac8b-74a0f45de889 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:05.613799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7a697489-67d9-42dd-ac8b-74a0f45de889 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:05.632877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a697489-67d9-42dd-ac8b-74a0f45de889 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:05.665701 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:05.675965 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:05.677392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:05.677943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] PATH_INFO: `/v3/projects/5efad2fec1c44d4a96661c9b4eb41610` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:05.680555 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5efad2fec1c44d4a96661c9b4eb41610)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:05.709813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:05.931388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5efad2fec1c44d4a96661c9b4eb41610', 'request_id': 'req-987f434f-3d04-4080-bad2-03c00eda1248'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:05.948041 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5efad2fec1c44d4a96661c9b4eb41610', 'request_id': 'req-987f434f-3d04-4080-bad2-03c00eda1248'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:05.965072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5efad2fec1c44d4a96661c9b4eb41610', 'request_id': 'req-987f434f-3d04-4080-bad2-03c00eda1248'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:06.007236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] The token cache is being invalidate because project 5efad2fec1c44d4a96661c9b4eb41610 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:12:06.019862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:06.021110 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:06.026595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1206/2414] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:12:05 2022] DELETE /identity/v3/projects/5efad2fec1c44d4a96661c9b4eb41610 => generated 0 bytes in 418 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:12:10.889543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbc50fc-f06b-4c1d-bfd3-fb6585cc74cf None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:10.890257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbc50fc-f06b-4c1d-bfd3-fb6585cc74cf None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:10.890640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbc50fc-f06b-4c1d-bfd3-fb6585cc74cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:10.928958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-3dbc50fc-f06b-4c1d-bfd3-fb6585cc74cf None None] MFA Rules not processed for user `1776c123f98c44b69807ec2b0281b718`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:10.930160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3dbc50fc-f06b-4c1d-bfd3-fb6585cc74cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:11.035633 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1209/2415] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:10 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:11.037113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:11.038949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:11.175026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987f434f-3d04-4080-bad2-03c00eda1248 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:11.183515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac67a6e-adbb-4400-af92-5c3a7468f86a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:11.184629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac67a6e-adbb-4400-af92-5c3a7468f86a admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:11.185117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac67a6e-adbb-4400-af92-5c3a7468f86a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:11.185630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac67a6e-adbb-4400-af92-5c3a7468f86a admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:11.186900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac67a6e-adbb-4400-af92-5c3a7468f86a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:11.190660 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac67a6e-adbb-4400-af92-5c3a7468f86a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:11.244812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1207/2416] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:12:11 2022] POST /identity/v3/projects => generated 385 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:12:11.249404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbc50fc-f06b-4c1d-bfd3-fb6585cc74cf None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:11.261656 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbc50fc-f06b-4c1d-bfd3-fb6585cc74cf None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:11.315286 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:11.328027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b279eca-365d-485d-a0c7-b4c37caab6ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:11.328700 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b279eca-365d-485d-a0c7-b4c37caab6ae admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:11.328975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b279eca-365d-485d-a0c7-b4c37caab6ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:11.329398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b279eca-365d-485d-a0c7-b4c37caab6ae admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:11.330000 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b279eca-365d-485d-a0c7-b4c37caab6ae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:11.332018 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b279eca-365d-485d-a0c7-b4c37caab6ae admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:11.359211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1210/2417] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:12:11 2022] POST /identity/v3/users => generated 299 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:12:11.364257 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac67a6e-adbb-4400-af92-5c3a7468f86a admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:11.386392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac67a6e-adbb-4400-af92-5c3a7468f86a admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:11.394609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93536ec6-48fa-437d-aaae-609f578db9fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:11.396960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93536ec6-48fa-437d-aaae-609f578db9fe admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:11.396960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93536ec6-48fa-437d-aaae-609f578db9fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:11.396960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93536ec6-48fa-437d-aaae-609f578db9fe admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:11.398593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93536ec6-48fa-437d-aaae-609f578db9fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:11.401796 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93536ec6-48fa-437d-aaae-609f578db9fe admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:11.421266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1208/2418] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:12:11 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:12:11.427553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b279eca-365d-485d-a0c7-b4c37caab6ae admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:11.455076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b279eca-365d-485d-a0c7-b4c37caab6ae admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:11.474970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f5ff4e-13d4-4190-946f-e0b2f6519cda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:11.476601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5ff4e-13d4-4190-946f-e0b2f6519cda admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:11.477260 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5ff4e-13d4-4190-946f-e0b2f6519cda admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:11.478132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5ff4e-13d4-4190-946f-e0b2f6519cda admin admin] PATH_INFO: `/v3/projects/771efa3d0e2b43e4a6394dbd92b85be1/users/025b941d2db04cd2b4fc018ed0fdf5b2/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:11.516549 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f5ff4e-13d4-4190-946f-e0b2f6519cda admin admin] RBAC: Authorizing `identity:create_grant(project_id=771efa3d0e2b43e4a6394dbd92b85be1, user_id=025b941d2db04cd2b4fc018ed0fdf5b2, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:11.531131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f5ff4e-13d4-4190-946f-e0b2f6519cda admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:11.573248 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1211/2419] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:12:11 2022] PUT /identity/v3/projects/771efa3d0e2b43e4a6394dbd92b85be1/users/025b941d2db04cd2b4fc018ed0fdf5b2/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:12:11.574507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93536ec6-48fa-437d-aaae-609f578db9fe admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:11.587283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93536ec6-48fa-437d-aaae-609f578db9fe admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:11.621314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:11.713020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16043818-37ed-4c13-865d-7b965fe00ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:11.719892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16043818-37ed-4c13-865d-7b965fe00ad7 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:11.720255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16043818-37ed-4c13-865d-7b965fe00ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:11.720698 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16043818-37ed-4c13-865d-7b965fe00ad7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:11.721987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16043818-37ed-4c13-865d-7b965fe00ad7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:11.725204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16043818-37ed-4c13-865d-7b965fe00ad7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:11.751125 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1209/2420] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:12:11 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:12:11.752699 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f5ff4e-13d4-4190-946f-e0b2f6519cda admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:11.793963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f5ff4e-13d4-4190-946f-e0b2f6519cda admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:11.799604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6143d0b-8895-4560-82b8-8e0a0c0cc1c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:11.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6143d0b-8895-4560-82b8-8e0a0c0cc1c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:11.800597 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6143d0b-8895-4560-82b8-8e0a0c0cc1c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:11.801011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6143d0b-8895-4560-82b8-8e0a0c0cc1c0 admin admin] PATH_INFO: `/v3/projects/771efa3d0e2b43e4a6394dbd92b85be1/users/025b941d2db04cd2b4fc018ed0fdf5b2/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:11.803851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6143d0b-8895-4560-82b8-8e0a0c0cc1c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=771efa3d0e2b43e4a6394dbd92b85be1, user_id=025b941d2db04cd2b4fc018ed0fdf5b2, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:11.807656 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6143d0b-8895-4560-82b8-8e0a0c0cc1c0 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:11.822313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1212/2421] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:12:11 2022] PUT /identity/v3/projects/771efa3d0e2b43e4a6394dbd92b85be1/users/025b941d2db04cd2b4fc018ed0fdf5b2/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:12:14.181167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6f9ceb-883f-4f66-ab16-9bf3c563218f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:14.182277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6f9ceb-883f-4f66-ab16-9bf3c563218f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:14.183622 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6f9ceb-883f-4f66-ab16-9bf3c563218f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:14.234003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-fd6f9ceb-883f-4f66-ab16-9bf3c563218f None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:14.236289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fd6f9ceb-883f-4f66-ab16-9bf3c563218f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:14.400126 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1210/2422] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:12:14 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:14.457605 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6143d0b-8895-4560-82b8-8e0a0c0cc1c0 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:14.460610 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d6143d0b-8895-4560-82b8-8e0a0c0cc1c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:14.652155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6143d0b-8895-4560-82b8-8e0a0c0cc1c0 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:14.658044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf62cef-c481-4c43-a5bc-44e9f487bee2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:14.658803 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf62cef-c481-4c43-a5bc-44e9f487bee2 service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:14.659144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf62cef-c481-4c43-a5bc-44e9f487bee2 service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:14.659489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf62cef-c481-4c43-a5bc-44e9f487bee2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:14.686564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf62cef-c481-4c43-a5bc-44e9f487bee2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:14.688520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf62cef-c481-4c43-a5bc-44e9f487bee2 service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:14.726941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1213/2423] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:12:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 31 01:12:21.116305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a9b582-5c68-453a-84ee-39e0c478a735 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:21.116847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a9b582-5c68-453a-84ee-39e0c478a735 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:21.116847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a9b582-5c68-453a-84ee-39e0c478a735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:21.177167 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-f5a9b582-5c68-453a-84ee-39e0c478a735 None None] MFA Rules not processed for user `025b941d2db04cd2b4fc018ed0fdf5b2`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:21.179390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f5a9b582-5c68-453a-84ee-39e0c478a735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:21.298926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1211/2424] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:21 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:21.308093 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf62cef-c481-4c43-a5bc-44e9f487bee2 service swift] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:21.309708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-bbf62cef-c481-4c43-a5bc-44e9f487bee2 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:21.401753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf62cef-c481-4c43-a5bc-44e9f487bee2 service swift] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:21.407411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79f7321-31a9-4c06-b038-db04709536ea service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4160039561434e51a87c6c7a019f2dff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:21.411336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79f7321-31a9-4c06-b038-db04709536ea service glance] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:21.411336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79f7321-31a9-4c06-b038-db04709536ea service glance] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:21.411336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79f7321-31a9-4c06-b038-db04709536ea service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:21.554410 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:21.621101 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79f7321-31a9-4c06-b038-db04709536ea service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:21.623264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79f7321-31a9-4c06-b038-db04709536ea service glance] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:21.638752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1214/2425] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:12:21 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:12:21.834862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d400c45c-c7fb-4f29-8bf1-cfb50523e09a None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:21.835426 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d400c45c-c7fb-4f29-8bf1-cfb50523e09a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:21.836741 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d400c45c-c7fb-4f29-8bf1-cfb50523e09a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:21.880713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d400c45c-c7fb-4f29-8bf1-cfb50523e09a None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:21.881937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d400c45c-c7fb-4f29-8bf1-cfb50523e09a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:21.898535 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1212/2426] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:12:21 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:21.910665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79f7321-31a9-4c06-b038-db04709536ea service glance] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:21.938063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79f7321-31a9-4c06-b038-db04709536ea service glance] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:21.947893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c17f65-ad44-491c-bf01-70f032b8a7ab service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:21.948926 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c17f65-ad44-491c-bf01-70f032b8a7ab service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:21.949422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c17f65-ad44-491c-bf01-70f032b8a7ab service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:21.949909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c17f65-ad44-491c-bf01-70f032b8a7ab service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:21.967659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c17f65-ad44-491c-bf01-70f032b8a7ab service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:21.970684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c17f65-ad44-491c-bf01-70f032b8a7ab service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:21.994232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1215/2427] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:12:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 31 01:12:28.963184 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8955a68f-370b-485a-aa79-afde2f2e6e8f None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:28.964473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8955a68f-370b-485a-aa79-afde2f2e6e8f None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:28.964995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8955a68f-370b-485a-aa79-afde2f2e6e8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:29.010817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-8955a68f-370b-485a-aa79-afde2f2e6e8f None None] MFA Rules not processed for user `52e5a4deed99442e8eed62f334aa670d`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:29.013942 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8955a68f-370b-485a-aa79-afde2f2e6e8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:29.130576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:29.163541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1213/2428] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:28 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:31.980334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=11 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:36.235338 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc6b7dd-2ce1-466b-bbab-570d59ef627a None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:36.236592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc6b7dd-2ce1-466b-bbab-570d59ef627a None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:36.236592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc6b7dd-2ce1-466b-bbab-570d59ef627a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:36.259551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-0dc6b7dd-2ce1-466b-bbab-570d59ef627a None None] MFA Rules not processed for user `1776c123f98c44b69807ec2b0281b718`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:36.261047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0dc6b7dd-2ce1-466b-bbab-570d59ef627a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:36.269853 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1216/2429] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:36 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:39.098043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08101e41-7e4b-47e8-a691-1e6b4a335c96 None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:39.098764 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08101e41-7e4b-47e8-a691-1e6b4a335c96 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:39.099040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08101e41-7e4b-47e8-a691-1e6b4a335c96 None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:39.101701 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1214/2430] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:12:39 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:12:39.122502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca68738d-c551-406d-826e-69d561d412cf None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:39.123078 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca68738d-c551-406d-826e-69d561d412cf None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:39.123238 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca68738d-c551-406d-826e-69d561d412cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:39.151896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:39.155624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-ca68738d-c551-406d-826e-69d561d412cf None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:39.157013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-ca68738d-c551-406d-826e-69d561d412cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:39.289319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1217/2431] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:12:39 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:12:39.302192 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08101e41-7e4b-47e8-a691-1e6b4a335c96 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:39.304043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-08101e41-7e4b-47e8-a691-1e6b4a335c96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:39.322407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08101e41-7e4b-47e8-a691-1e6b4a335c96 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:39.328788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5569b9de-ce7d-46e6-bef1-b4857eb7aa4e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4160039561434e51a87c6c7a019f2dff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:39.329815 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5569b9de-ce7d-46e6-bef1-b4857eb7aa4e service glance] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:39.330289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5569b9de-ce7d-46e6-bef1-b4857eb7aa4e service glance] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:39.330993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5569b9de-ce7d-46e6-bef1-b4857eb7aa4e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:39.350364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5569b9de-ce7d-46e6-bef1-b4857eb7aa4e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:39.353322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5569b9de-ce7d-46e6-bef1-b4857eb7aa4e service glance] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:39.374368 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1215/2432] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:12:39 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:12:39.453272 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e03b60-f7cb-49da-996f-5e0d0be833dd None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:39.453592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e03b60-f7cb-49da-996f-5e0d0be833dd None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:39.453865 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e03b60-f7cb-49da-996f-5e0d0be833dd None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:39.455313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1218/2433] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:12:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:12:39.461906 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a521d4d5-1498-44f2-81fe-514e2275da54 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:39.462266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a521d4d5-1498-44f2-81fe-514e2275da54 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:39.462617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a521d4d5-1498-44f2-81fe-514e2275da54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:39.484936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a521d4d5-1498-44f2-81fe-514e2275da54 None None] MFA Rules not processed for user `94e868131d33472b9ded8486274a07b9`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:39.486470 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a521d4d5-1498-44f2-81fe-514e2275da54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:39.544965 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1216/2434] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:12:39 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:39.558372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e03b60-f7cb-49da-996f-5e0d0be833dd None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:39.560352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-43e03b60-f7cb-49da-996f-5e0d0be833dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:39.575059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e03b60-f7cb-49da-996f-5e0d0be833dd None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:39.579480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd51c2f4-5650-4aae-b8ac-d9ae16046568 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:39.580235 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd51c2f4-5650-4aae-b8ac-d9ae16046568 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:39.580552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd51c2f4-5650-4aae-b8ac-d9ae16046568 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:39.580822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd51c2f4-5650-4aae-b8ac-d9ae16046568 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:39.589970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd51c2f4-5650-4aae-b8ac-d9ae16046568 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:39.591495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd51c2f4-5650-4aae-b8ac-d9ae16046568 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:39.600715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1219/2435] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:12:39 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:12:39.703092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03645071-2f3e-4a6e-b77e-803f1c64008a None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:39.703522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03645071-2f3e-4a6e-b77e-803f1c64008a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:39.703954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03645071-2f3e-4a6e-b77e-803f1c64008a None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:39.705587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1217/2436] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:12:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 01:12:39.712331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b32988b-6123-459e-b382-828675f347eb None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:39.712616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b32988b-6123-459e-b382-828675f347eb None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:39.712898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b32988b-6123-459e-b382-828675f347eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:39.733942 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-7b32988b-6123-459e-b382-828675f347eb None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:39.735196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7b32988b-6123-459e-b382-828675f347eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:39.746162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1220/2437] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:12:39 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:39.758751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03645071-2f3e-4a6e-b77e-803f1c64008a None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:39.762254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-03645071-2f3e-4a6e-b77e-803f1c64008a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:39.787519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03645071-2f3e-4a6e-b77e-803f1c64008a None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:39.794648 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f87fccd-703e-4df9-b05d-16fa1d47bacd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:39.796205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f87fccd-703e-4df9-b05d-16fa1d47bacd service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:39.797056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f87fccd-703e-4df9-b05d-16fa1d47bacd service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:39.797408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f87fccd-703e-4df9-b05d-16fa1d47bacd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:39.821458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f87fccd-703e-4df9-b05d-16fa1d47bacd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:39.823435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f87fccd-703e-4df9-b05d-16fa1d47bacd service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:39.843632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1218/2438] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:12:39 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:12:42.970630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d2263a-b156-436d-94d7-782b4c6fb5b9 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:42.971312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d2263a-b156-436d-94d7-782b4c6fb5b9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:42.971749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d2263a-b156-436d-94d7-782b4c6fb5b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:43.042980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-61d2263a-b156-436d-94d7-782b4c6fb5b9 None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:43.060044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-61d2263a-b156-436d-94d7-782b4c6fb5b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:43.079311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1221/2439] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:12:42 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:43.106716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f87fccd-703e-4df9-b05d-16fa1d47bacd service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:43.162810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f87fccd-703e-4df9-b05d-16fa1d47bacd service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:43.194375 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1986e542-cacf-4738-8304-d69bd8b60498 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:43.195730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1986e542-cacf-4738-8304-d69bd8b60498 service swift] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:43.196212 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1986e542-cacf-4738-8304-d69bd8b60498 service swift] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:43.196650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1986e542-cacf-4738-8304-d69bd8b60498 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:43.280770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1986e542-cacf-4738-8304-d69bd8b60498 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:43.284381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1986e542-cacf-4738-8304-d69bd8b60498 service swift] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:43.309741 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1219/2440] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:12:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 01:12:48.750570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d2263a-b156-436d-94d7-782b4c6fb5b9 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:48.752994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-61d2263a-b156-436d-94d7-782b4c6fb5b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:48.909837 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d2263a-b156-436d-94d7-782b4c6fb5b9 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:48.920454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:48.921312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:48.921718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:48.922113 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] PATH_INFO: `/v3/users/b4d5926ba9644db5bda5c7feeaf5d60c` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:48.923982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] RBAC: Authorizing `identity:delete_user(user_id=b4d5926ba9644db5bda5c7feeaf5d60c)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:48.927914 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:49.049211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b4d5926ba9644db5bda5c7feeaf5d60c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:49.049921 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b4d5926ba9644db5bda5c7feeaf5d60c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:49.075211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b4d5926ba9644db5bda5c7feeaf5d60c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:49.091532 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:49.104785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b4d5926ba9644db5bda5c7feeaf5d60c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:49.128805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b4d5926ba9644db5bda5c7feeaf5d60c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:49.137857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1222/2441] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:12:48 2022] DELETE /identity/v3/users/b4d5926ba9644db5bda5c7feeaf5d60c => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:12:49.143667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa1a183-c8b6-4c83-85a7-c81bb797ac49 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:49.145104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa1a183-c8b6-4c83-85a7-c81bb797ac49 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:49.145841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa1a183-c8b6-4c83-85a7-c81bb797ac49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:49.228558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-5aa1a183-c8b6-4c83-85a7-c81bb797ac49 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:49.231398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5aa1a183-c8b6-4c83-85a7-c81bb797ac49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:49.403590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1220/2442] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:49.417099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:49.418851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:49.552936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a7d15c-e579-4c98-a8b4-9c822bcaef3e admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:49.558442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10288cf-a63e-4b63-a4bf-eeef395cf359 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:49.559404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10288cf-a63e-4b63-a4bf-eeef395cf359 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:49.559857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10288cf-a63e-4b63-a4bf-eeef395cf359 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:49.560277 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10288cf-a63e-4b63-a4bf-eeef395cf359 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:49.585747 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10288cf-a63e-4b63-a4bf-eeef395cf359 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:49.589553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10288cf-a63e-4b63-a4bf-eeef395cf359 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:49.609624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1223/2443] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:12:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:12:50.314950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa1a183-c8b6-4c83-85a7-c81bb797ac49 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:50.317189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5aa1a183-c8b6-4c83-85a7-c81bb797ac49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:50.342625 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa1a183-c8b6-4c83-85a7-c81bb797ac49 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:50.350001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbe41de-a4cb-4b17-8811-422f1f76cd76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:50.351175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbe41de-a4cb-4b17-8811-422f1f76cd76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:50.351508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbe41de-a4cb-4b17-8811-422f1f76cd76 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:50.351790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbe41de-a4cb-4b17-8811-422f1f76cd76 admin admin] PATH_INFO: `/v3/projects/d25c5c4a187645eaa71c6f2015af6b57` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:50.367531 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbe41de-a4cb-4b17-8811-422f1f76cd76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d25c5c4a187645eaa71c6f2015af6b57)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:50.373237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbe41de-a4cb-4b17-8811-422f1f76cd76 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:50.509932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-adbe41de-a4cb-4b17-8811-422f1f76cd76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd25c5c4a187645eaa71c6f2015af6b57', 'request_id': 'req-adbe41de-a4cb-4b17-8811-422f1f76cd76'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:50.535848 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-adbe41de-a4cb-4b17-8811-422f1f76cd76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd25c5c4a187645eaa71c6f2015af6b57', 'request_id': 'req-adbe41de-a4cb-4b17-8811-422f1f76cd76'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:50.577856 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-adbe41de-a4cb-4b17-8811-422f1f76cd76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd25c5c4a187645eaa71c6f2015af6b57', 'request_id': 'req-adbe41de-a4cb-4b17-8811-422f1f76cd76'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:50.589993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:50.625194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-adbe41de-a4cb-4b17-8811-422f1f76cd76 admin admin] The token cache is being invalidate because project d25c5c4a187645eaa71c6f2015af6b57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:12:50.625712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-adbe41de-a4cb-4b17-8811-422f1f76cd76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:50.626903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-adbe41de-a4cb-4b17-8811-422f1f76cd76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:12:50.632163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1221/2444] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:12:50 2022] DELETE /identity/v3/projects/d25c5c4a187645eaa71c6f2015af6b57 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:12:50.639771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baec4b4-cb2b-41eb-b3ac-70657cd1f419 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:50.641809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baec4b4-cb2b-41eb-b3ac-70657cd1f419 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:50.641809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baec4b4-cb2b-41eb-b3ac-70657cd1f419 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:50.748612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-3baec4b4-cb2b-41eb-b3ac-70657cd1f419 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:50.750963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3baec4b4-cb2b-41eb-b3ac-70657cd1f419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:50.953362 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1224/2445] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:12:50.956437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e929c02-6662-4ad1-9840-9e98831fd91a None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:50.956808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e929c02-6662-4ad1-9840-9e98831fd91a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:50.957145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e929c02-6662-4ad1-9840-9e98831fd91a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:51.026345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-0e929c02-6662-4ad1-9840-9e98831fd91a None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:51.030064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0e929c02-6662-4ad1-9840-9e98831fd91a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:51.040729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1222/2446] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:51.048034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baec4b4-cb2b-41eb-b3ac-70657cd1f419 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:51.084944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baec4b4-cb2b-41eb-b3ac-70657cd1f419 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:51.093647 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d213f6c-2972-4658-ae16-60b5e345438e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:51.094708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d213f6c-2972-4658-ae16-60b5e345438e admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:51.095271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d213f6c-2972-4658-ae16-60b5e345438e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:51.095771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d213f6c-2972-4658-ae16-60b5e345438e admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:51.097522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d213f6c-2972-4658-ae16-60b5e345438e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:12:51.098068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d213f6c-2972-4658-ae16-60b5e345438e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:51.101292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d213f6c-2972-4658-ae16-60b5e345438e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:51.123567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1225/2447] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:12:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:12:51.130255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e929c02-6662-4ad1-9840-9e98831fd91a None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:51.149453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e929c02-6662-4ad1-9840-9e98831fd91a None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:51.157239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4629744c-32d9-48ba-8938-1aa5dbac69e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:51.158252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4629744c-32d9-48ba-8938-1aa5dbac69e1 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:51.158725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4629744c-32d9-48ba-8938-1aa5dbac69e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:51.159510 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4629744c-32d9-48ba-8938-1aa5dbac69e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:51.161035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4629744c-32d9-48ba-8938-1aa5dbac69e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:51.166040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4629744c-32d9-48ba-8938-1aa5dbac69e1 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:51.211048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1223/2448] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:12:51 2022] POST /identity/v3/projects => generated 371 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:12:51.216325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d213f6c-2972-4658-ae16-60b5e345438e admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:51.264845 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d213f6c-2972-4658-ae16-60b5e345438e admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:51.391915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10591671-3f1e-41d9-b1de-f2c5c1d7829e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:51.392912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10591671-3f1e-41d9-b1de-f2c5c1d7829e admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:51.393417 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10591671-3f1e-41d9-b1de-f2c5c1d7829e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:51.393966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10591671-3f1e-41d9-b1de-f2c5c1d7829e admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:51.395256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10591671-3f1e-41d9-b1de-f2c5c1d7829e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:51.399812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10591671-3f1e-41d9-b1de-f2c5c1d7829e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:51.433600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:12:51.448578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1226/2449] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:12:51 2022] POST /identity/v3/users => generated 286 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:12:51.449834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4629744c-32d9-48ba-8938-1aa5dbac69e1 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:51.479284 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4629744c-32d9-48ba-8938-1aa5dbac69e1 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:51.488784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4681e6d-5802-4e6d-bf17-e61008a820db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:51.489849 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4681e6d-5802-4e6d-bf17-e61008a820db admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:51.490355 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4681e6d-5802-4e6d-bf17-e61008a820db admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:51.490947 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4681e6d-5802-4e6d-bf17-e61008a820db admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:51.492527 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4681e6d-5802-4e6d-bf17-e61008a820db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:51.495237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4681e6d-5802-4e6d-bf17-e61008a820db admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:51.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1224/2450] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:12:51 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:12:51.523525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10591671-3f1e-41d9-b1de-f2c5c1d7829e admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:51.546961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10591671-3f1e-41d9-b1de-f2c5c1d7829e admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:51.558244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6fad1c-bda7-4ec7-b672-b06c92c20769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:51.559233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6fad1c-bda7-4ec7-b672-b06c92c20769 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:51.559852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6fad1c-bda7-4ec7-b672-b06c92c20769 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:51.560354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6fad1c-bda7-4ec7-b672-b06c92c20769 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:51.561295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6fad1c-bda7-4ec7-b672-b06c92c20769 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:51.563271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6fad1c-bda7-4ec7-b672-b06c92c20769 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:51.596405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1227/2451] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:12:51 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:12:51.598142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4681e6d-5802-4e6d-bf17-e61008a820db admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:51.613231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4681e6d-5802-4e6d-bf17-e61008a820db admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:51.622000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968d5292-8a28-4909-b917-45c8b8e7a882 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:51.623229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968d5292-8a28-4909-b917-45c8b8e7a882 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:51.623784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968d5292-8a28-4909-b917-45c8b8e7a882 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:51.624412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968d5292-8a28-4909-b917-45c8b8e7a882 admin admin] PATH_INFO: `/v3/projects/5ca9597665cd4068b0a111136d6e0489/users/cb4bfd7ea6664747a6e0693a2fd11e3b/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:51.675896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968d5292-8a28-4909-b917-45c8b8e7a882 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ca9597665cd4068b0a111136d6e0489, user_id=cb4bfd7ea6664747a6e0693a2fd11e3b, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:51.682844 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968d5292-8a28-4909-b917-45c8b8e7a882 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:51.717322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1225/2452] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:12:51 2022] PUT /identity/v3/projects/5ca9597665cd4068b0a111136d6e0489/users/cb4bfd7ea6664747a6e0693a2fd11e3b/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:12:51.722859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d583fd5-9849-4e91-b1b6-995ec189ac7e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:51.723576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d583fd5-9849-4e91-b1b6-995ec189ac7e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:51.723576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d583fd5-9849-4e91-b1b6-995ec189ac7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:51.772343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-1d583fd5-9849-4e91-b1b6-995ec189ac7e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:12:51.775128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-1d583fd5-9849-4e91-b1b6-995ec189ac7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:51.889317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1228/2453] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:12:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:12:51.902056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968d5292-8a28-4909-b917-45c8b8e7a882 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:12:51.905567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-968d5292-8a28-4909-b917-45c8b8e7a882 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:12:52.051277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968d5292-8a28-4909-b917-45c8b8e7a882 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:12:52.067615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83f8fa3-3fc1-4668-803c-f49c179d86d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:12:52.068630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83f8fa3-3fc1-4668-803c-f49c179d86d2 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:12:52.069145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83f8fa3-3fc1-4668-803c-f49c179d86d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:12:52.070139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83f8fa3-3fc1-4668-803c-f49c179d86d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:12:52.108287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83f8fa3-3fc1-4668-803c-f49c179d86d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:12:52.112100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83f8fa3-3fc1-4668-803c-f49c179d86d2 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:12:52.150107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1226/2454] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:12:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:13:00.527399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad7fcda-7e21-429e-9fd9-8fa489cb7692 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:00.528469 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad7fcda-7e21-429e-9fd9-8fa489cb7692 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:00.530252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad7fcda-7e21-429e-9fd9-8fa489cb7692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:00.599502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-0ad7fcda-7e21-429e-9fd9-8fa489cb7692 None None] MFA Rules not processed for user `cb4bfd7ea6664747a6e0693a2fd11e3b`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:13:00.602461 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0ad7fcda-7e21-429e-9fd9-8fa489cb7692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:00.730957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1229/2455] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:13:00 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:13:00.737321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83f8fa3-3fc1-4668-803c-f49c179d86d2 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:00.758176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83f8fa3-3fc1-4668-803c-f49c179d86d2 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:00.769963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885eaa67-8aff-45b3-b263-8b35b44baa3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:00.772297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885eaa67-8aff-45b3-b263-8b35b44baa3c admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:00.773088 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885eaa67-8aff-45b3-b263-8b35b44baa3c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:00.774877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885eaa67-8aff-45b3-b263-8b35b44baa3c admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:00.778291 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885eaa67-8aff-45b3-b263-8b35b44baa3c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:00.785506 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885eaa67-8aff-45b3-b263-8b35b44baa3c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:00.839078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1227/2456] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:13:00 2022] POST /identity/v3/projects => generated 371 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:13:00.844480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad7fcda-7e21-429e-9fd9-8fa489cb7692 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:00.863667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad7fcda-7e21-429e-9fd9-8fa489cb7692 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:00.937443 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbefa793-315c-4649-b0fc-d8c8674a683c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:00.938297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbefa793-315c-4649-b0fc-d8c8674a683c admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:00.938642 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbefa793-315c-4649-b0fc-d8c8674a683c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:00.939171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbefa793-315c-4649-b0fc-d8c8674a683c admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:00.939967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbefa793-315c-4649-b0fc-d8c8674a683c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:00.942475 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbefa793-315c-4649-b0fc-d8c8674a683c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:00.971319 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1230/2457] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:13:00 2022] POST /identity/v3/users => generated 292 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:13:00.975727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885eaa67-8aff-45b3-b263-8b35b44baa3c admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:00.988122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885eaa67-8aff-45b3-b263-8b35b44baa3c admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:00.994435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616048b3-60ca-4fe8-bccf-c25ccbaaf3ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:00.995190 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616048b3-60ca-4fe8-bccf-c25ccbaaf3ed admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:00.995587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616048b3-60ca-4fe8-bccf-c25ccbaaf3ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:00.996142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616048b3-60ca-4fe8-bccf-c25ccbaaf3ed admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:00.996928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616048b3-60ca-4fe8-bccf-c25ccbaaf3ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:00.998759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616048b3-60ca-4fe8-bccf-c25ccbaaf3ed admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:01.011770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1228/2458] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:13:00 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:13:01.016633 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbefa793-315c-4649-b0fc-d8c8674a683c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:01.028113 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbefa793-315c-4649-b0fc-d8c8674a683c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:01.034944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c278d2c-14c3-4f56-abaf-7d52ac4b1d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:01.035890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c278d2c-14c3-4f56-abaf-7d52ac4b1d06 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:01.036359 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c278d2c-14c3-4f56-abaf-7d52ac4b1d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:01.036818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c278d2c-14c3-4f56-abaf-7d52ac4b1d06 admin admin] PATH_INFO: `/v3/projects/1ebe85a1d3e9405fb63f4ce48c29efa6/users/002e3e2e5b7740c58c65072e11a80e51/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:01.040195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:13:01.060819 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c278d2c-14c3-4f56-abaf-7d52ac4b1d06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ebe85a1d3e9405fb63f4ce48c29efa6, user_id=002e3e2e5b7740c58c65072e11a80e51, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:01.066631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c278d2c-14c3-4f56-abaf-7d52ac4b1d06 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:01.094368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1231/2459] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:13:01 2022] PUT /identity/v3/projects/1ebe85a1d3e9405fb63f4ce48c29efa6/users/002e3e2e5b7740c58c65072e11a80e51/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:13:01.096514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616048b3-60ca-4fe8-bccf-c25ccbaaf3ed admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:01.118542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616048b3-60ca-4fe8-bccf-c25ccbaaf3ed admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:01.216690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1190c6d3-54ef-4660-9c88-4df6d21dcc2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:01.217763 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1190c6d3-54ef-4660-9c88-4df6d21dcc2e admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:01.219174 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1190c6d3-54ef-4660-9c88-4df6d21dcc2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:01.219935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1190c6d3-54ef-4660-9c88-4df6d21dcc2e admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:01.221874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1190c6d3-54ef-4660-9c88-4df6d21dcc2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:01.224669 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1190c6d3-54ef-4660-9c88-4df6d21dcc2e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:01.227888 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:13:01.246685 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1229/2460] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:13:01 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:13:01.251707 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c278d2c-14c3-4f56-abaf-7d52ac4b1d06 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:01.267580 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c278d2c-14c3-4f56-abaf-7d52ac4b1d06 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:01.281368 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4064051-281f-462d-bc1d-8857d86aeae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:01.282894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4064051-281f-462d-bc1d-8857d86aeae5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:01.283762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4064051-281f-462d-bc1d-8857d86aeae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:01.284343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4064051-281f-462d-bc1d-8857d86aeae5 admin admin] PATH_INFO: `/v3/projects/1ebe85a1d3e9405fb63f4ce48c29efa6/users/002e3e2e5b7740c58c65072e11a80e51/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:01.288659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4064051-281f-462d-bc1d-8857d86aeae5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ebe85a1d3e9405fb63f4ce48c29efa6, user_id=002e3e2e5b7740c58c65072e11a80e51, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:01.294135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4064051-281f-462d-bc1d-8857d86aeae5 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:01.310196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1232/2461] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:13:01 2022] PUT /identity/v3/projects/1ebe85a1d3e9405fb63f4ce48c29efa6/users/002e3e2e5b7740c58c65072e11a80e51/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:13:04.005958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09dc755-e517-4d14-bc55-bfb061aa9a0c None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:04.006834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09dc755-e517-4d14-bc55-bfb061aa9a0c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:04.008206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09dc755-e517-4d14-bc55-bfb061aa9a0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:04.081998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-d09dc755-e517-4d14-bc55-bfb061aa9a0c None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:13:04.085406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d09dc755-e517-4d14-bc55-bfb061aa9a0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:04.315517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1230/2462] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:13:04 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 310 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:13:04.329180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4064051-281f-462d-bc1d-8857d86aeae5 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:04.331439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e4064051-281f-462d-bc1d-8857d86aeae5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:04.417326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09dc755-e517-4d14-bc55-bfb061aa9a0c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:04.419031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d09dc755-e517-4d14-bc55-bfb061aa9a0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:04.532341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4064051-281f-462d-bc1d-8857d86aeae5 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:04.542035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea96bee-a1b3-4960-b521-d110bea7c865 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:04.542806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea96bee-a1b3-4960-b521-d110bea7c865 service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:04.543199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea96bee-a1b3-4960-b521-d110bea7c865 service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:04.543603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea96bee-a1b3-4960-b521-d110bea7c865 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:04.565140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea96bee-a1b3-4960-b521-d110bea7c865 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:04.567109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea96bee-a1b3-4960-b521-d110bea7c865 service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:04.591109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1233/2463] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:13:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 01:13:04.626992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09dc755-e517-4d14-bc55-bfb061aa9a0c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:04.635900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0e2301-5af0-4239-819f-2fc5e8d6e790 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:04.636961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e2301-5af0-4239-819f-2fc5e8d6e790 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:04.637451 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e2301-5af0-4239-819f-2fc5e8d6e790 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:04.637934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e2301-5af0-4239-819f-2fc5e8d6e790 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:04.646360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-3b0e2301-5af0-4239-819f-2fc5e8d6e790 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:05.022403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e2301-5af0-4239-819f-2fc5e8d6e790 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:05.038034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e2301-5af0-4239-819f-2fc5e8d6e790 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:05.083957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1231/2464] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:13:04 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 667 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:13:06.988484 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea96bee-a1b3-4960-b521-d110bea7c865 service swift] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:07.004012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea96bee-a1b3-4960-b521-d110bea7c865 service swift] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:07.013973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524f8253-3d18-4fe6-a48a-25e760dab796 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:07.014704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f8253-3d18-4fe6-a48a-25e760dab796 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:07.015066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f8253-3d18-4fe6-a48a-25e760dab796 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:07.018025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524f8253-3d18-4fe6-a48a-25e760dab796 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:07.022598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-524f8253-3d18-4fe6-a48a-25e760dab796 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:07.046409 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524f8253-3d18-4fe6-a48a-25e760dab796 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:07.048352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524f8253-3d18-4fe6-a48a-25e760dab796 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:07.067745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1234/2465] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:13:06 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:13:12.528615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2d4c94-891f-4524-9850-11c55b7a18c5 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:12.529201 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2d4c94-891f-4524-9850-11c55b7a18c5 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:12.529511 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2d4c94-891f-4524-9850-11c55b7a18c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:12.676672 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-1d2d4c94-891f-4524-9850-11c55b7a18c5 None None] MFA Rules not processed for user `002e3e2e5b7740c58c65072e11a80e51`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:13:12.687438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1d2d4c94-891f-4524-9850-11c55b7a18c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:12.785739 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:13:12.896241 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1232/2466] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:13:12 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 369 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:13:12.967726 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524f8253-3d18-4fe6-a48a-25e760dab796 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:12.971666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-524f8253-3d18-4fe6-a48a-25e760dab796 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:13.003226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524f8253-3d18-4fe6-a48a-25e760dab796 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:13.024694 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6300676d-e91f-415a-96fc-017d25c04f96 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:13.025730 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6300676d-e91f-415a-96fc-017d25c04f96 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:13.026289 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6300676d-e91f-415a-96fc-017d25c04f96 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:13.026789 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6300676d-e91f-415a-96fc-017d25c04f96 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:13.164314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6300676d-e91f-415a-96fc-017d25c04f96 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:13.166846 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6300676d-e91f-415a-96fc-017d25c04f96 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:13.183949 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1235/2467] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:13:12 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:13:15.711667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a8b11d-8e86-40d5-aca7-8daf6b86ae75 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:15.712279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a8b11d-8e86-40d5-aca7-8daf6b86ae75 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:15.712640 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a8b11d-8e86-40d5-aca7-8daf6b86ae75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:15.788977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b3a8b11d-8e86-40d5-aca7-8daf6b86ae75 None None] MFA Rules not processed for user `cb4bfd7ea6664747a6e0693a2fd11e3b`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:13:15.797677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b3a8b11d-8e86-40d5-aca7-8daf6b86ae75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:15.827883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1233/2468] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:13:15 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:13:22.547524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6300676d-e91f-415a-96fc-017d25c04f96 service nova] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:22.549893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6300676d-e91f-415a-96fc-017d25c04f96 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:22.577234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6300676d-e91f-415a-96fc-017d25c04f96 service nova] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:22.586653 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9698a34-3e2e-450a-926d-edd8c9772308 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:22.587771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9698a34-3e2e-450a-926d-edd8c9772308 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:22.588274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9698a34-3e2e-450a-926d-edd8c9772308 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:22.588781 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9698a34-3e2e-450a-926d-edd8c9772308 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:22.591867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e9698a34-3e2e-450a-926d-edd8c9772308 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:22.649442 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:13:22.675309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9698a34-3e2e-450a-926d-edd8c9772308 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:22.677603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9698a34-3e2e-450a-926d-edd8c9772308 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:22.689969 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1236/2469] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:13:22 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:13:25.774073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=18 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:13:32.684555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=8 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:13:42.689040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1e9410-accf-4de0-9a87-c0233a8d26da None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:42.689892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1e9410-accf-4de0-9a87-c0233a8d26da None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:42.689892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1e9410-accf-4de0-9a87-c0233a8d26da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:42.724069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-9e1e9410-accf-4de0-9a87-c0233a8d26da None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:13:42.725652 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9e1e9410-accf-4de0-9a87-c0233a8d26da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:42.733967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1234/2470] 10.222.0.42 () {58 vars in 967 bytes} [Sun Jul 31 01:13:42 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:13:42.763077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9698a34-3e2e-450a-926d-edd8c9772308 service nova] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:42.777292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9698a34-3e2e-450a-926d-edd8c9772308 service nova] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:42.783927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7cad2d-18df-4b57-bd70-93b832f27cbd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:42.784900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7cad2d-18df-4b57-bd70-93b832f27cbd service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:42.785171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7cad2d-18df-4b57-bd70-93b832f27cbd service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:42.785515 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7cad2d-18df-4b57-bd70-93b832f27cbd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:42.795834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7cad2d-18df-4b57-bd70-93b832f27cbd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:42.797796 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7cad2d-18df-4b57-bd70-93b832f27cbd service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:42.808400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1237/2471] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:13:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 01:13:43.170606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1e9410-accf-4de0-9a87-c0233a8d26da None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:43.193290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1e9410-accf-4de0-9a87-c0233a8d26da None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:43.199833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfd72fc-495c-4e7b-ba2a-103cae15928a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:43.200909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfd72fc-495c-4e7b-ba2a-103cae15928a service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:43.201419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfd72fc-495c-4e7b-ba2a-103cae15928a service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:43.201922 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfd72fc-495c-4e7b-ba2a-103cae15928a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:43.205278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ddfd72fc-495c-4e7b-ba2a-103cae15928a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:43.224834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfd72fc-495c-4e7b-ba2a-103cae15928a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:43.227744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfd72fc-495c-4e7b-ba2a-103cae15928a service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:43.249111 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1235/2472] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:13:43 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:13:43.960941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7cad2d-18df-4b57-bd70-93b832f27cbd service swift] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:43.963850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2b7cad2d-18df-4b57-bd70-93b832f27cbd service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:44.100254 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7cad2d-18df-4b57-bd70-93b832f27cbd service swift] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:44.114756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3422cc6e-0335-4144-b282-07d52bcce672 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:44.116268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3422cc6e-0335-4144-b282-07d52bcce672 service placement] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:44.116760 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3422cc6e-0335-4144-b282-07d52bcce672 service placement] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:44.117314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3422cc6e-0335-4144-b282-07d52bcce672 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:44.121706 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3422cc6e-0335-4144-b282-07d52bcce672 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:44.147861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3422cc6e-0335-4144-b282-07d52bcce672 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:44.151180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3422cc6e-0335-4144-b282-07d52bcce672 service placement] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:44.173986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1238/2473] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:13:43 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:13:46.222620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfd72fc-495c-4e7b-ba2a-103cae15928a service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:46.242238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfd72fc-495c-4e7b-ba2a-103cae15928a service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:46.251010 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a91272-ddb5-4a64-865d-e97e5cf613b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:46.252081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a91272-ddb5-4a64-865d-e97e5cf613b1 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:46.252562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a91272-ddb5-4a64-865d-e97e5cf613b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:46.253056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a91272-ddb5-4a64-865d-e97e5cf613b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:46.257048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e7a91272-ddb5-4a64-865d-e97e5cf613b1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:46.352307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a91272-ddb5-4a64-865d-e97e5cf613b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:46.354730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a91272-ddb5-4a64-865d-e97e5cf613b1 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:46.372457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1236/2474] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:13:46 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:13:46.834744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3422cc6e-0335-4144-b282-07d52bcce672 service placement] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:46.863923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3422cc6e-0335-4144-b282-07d52bcce672 service placement] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:46.874916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9765161e-f579-4adc-a679-19b6281b172a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:46.876499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9765161e-f579-4adc-a679-19b6281b172a service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:46.877318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9765161e-f579-4adc-a679-19b6281b172a service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:46.878025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9765161e-f579-4adc-a679-19b6281b172a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:46.883669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9765161e-f579-4adc-a679-19b6281b172a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:46.900827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9765161e-f579-4adc-a679-19b6281b172a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:46.902553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9765161e-f579-4adc-a679-19b6281b172a service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:46.917778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1239/2475] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:13:46 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:13:49.070324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a91272-ddb5-4a64-865d-e97e5cf613b1 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:13:49.099412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a91272-ddb5-4a64-865d-e97e5cf613b1 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:13:49.107441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e172598-3b5c-4fc1-bc33-93a5ff90eb7a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:13:49.107979 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e172598-3b5c-4fc1-bc33-93a5ff90eb7a service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:13:49.108270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e172598-3b5c-4fc1-bc33-93a5ff90eb7a service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:13:49.108609 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e172598-3b5c-4fc1-bc33-93a5ff90eb7a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:13:49.110943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6e172598-3b5c-4fc1-bc33-93a5ff90eb7a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:13:49.126604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e172598-3b5c-4fc1-bc33-93a5ff90eb7a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:13:49.129010 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e172598-3b5c-4fc1-bc33-93a5ff90eb7a service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:13:49.142780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1237/2476] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:13:49 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:13:56.912792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=44 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:13:59.139155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=45 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:14:01.795437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9765161e-f579-4adc-a679-19b6281b172a service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:14:01.810222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9765161e-f579-4adc-a679-19b6281b172a service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:14:01.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cb6a41-c97a-4029-bec4-75b5d1031525 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:14:01.817600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cb6a41-c97a-4029-bec4-75b5d1031525 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:14:01.817951 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cb6a41-c97a-4029-bec4-75b5d1031525 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:14:01.818230 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cb6a41-c97a-4029-bec4-75b5d1031525 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:14:01.820213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-86cb6a41-c97a-4029-bec4-75b5d1031525 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:14:01.891219 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86cb6a41-c97a-4029-bec4-75b5d1031525 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:14:01.893007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86cb6a41-c97a-4029-bec4-75b5d1031525 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:14:01.904567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1240/2477] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:14:01 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:14:02.830222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e172598-3b5c-4fc1-bc33-93a5ff90eb7a service nova] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:14:02.840921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e172598-3b5c-4fc1-bc33-93a5ff90eb7a service nova] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:14:02.845039 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc41ca33-7bf3-4177-ab7d-e3763ceb2fae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:14:02.845586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc41ca33-7bf3-4177-ab7d-e3763ceb2fae service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:14:02.845936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc41ca33-7bf3-4177-ab7d-e3763ceb2fae service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:14:02.846143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc41ca33-7bf3-4177-ab7d-e3763ceb2fae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:14:02.858547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc41ca33-7bf3-4177-ab7d-e3763ceb2fae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:14:02.860668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc41ca33-7bf3-4177-ab7d-e3763ceb2fae service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:14:02.873301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1238/2478] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:14:02 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:14:05.995592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cb6a41-c97a-4029-bec4-75b5d1031525 service nova] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:14:06.017468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cb6a41-c97a-4029-bec4-75b5d1031525 service nova] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:14:06.028445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3498d59f-71b7-4fa4-8fd8-267ea6016f9f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:14:06.029511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3498d59f-71b7-4fa4-8fd8-267ea6016f9f service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:14:06.030014 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3498d59f-71b7-4fa4-8fd8-267ea6016f9f service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:14:06.030520 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3498d59f-71b7-4fa4-8fd8-267ea6016f9f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:14:06.033945 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3498d59f-71b7-4fa4-8fd8-267ea6016f9f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:14:06.062750 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3498d59f-71b7-4fa4-8fd8-267ea6016f9f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:14:06.066737 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3498d59f-71b7-4fa4-8fd8-267ea6016f9f service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:14:06.090890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1241/2479] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:14:05 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:14:12.868396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=6 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:14:16.080415 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=34 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:14:17.366131 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a903e-1717-4783-ae84-7161a8327bed None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:14:17.366960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a903e-1717-4783-ae84-7161a8327bed None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:14:17.366960 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a903e-1717-4783-ae84-7161a8327bed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:14:17.390776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-046a903e-1717-4783-ae84-7161a8327bed None None] MFA Rules not processed for user `20df8d74c87143dd81dbaae76b3c2b62`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:14:17.392270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-046a903e-1717-4783-ae84-7161a8327bed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:14:17.401863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1239/2480] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:14:17 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:14:17.835413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3498d59f-71b7-4fa4-8fd8-267ea6016f9f service nova] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:14:17.845959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3498d59f-71b7-4fa4-8fd8-267ea6016f9f service nova] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:14:17.850570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b52562-9d6b-4524-875a-de09018ff46d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:14:17.851205 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b52562-9d6b-4524-875a-de09018ff46d service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:14:17.851521 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b52562-9d6b-4524-875a-de09018ff46d service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:14:17.851812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b52562-9d6b-4524-875a-de09018ff46d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:14:17.853839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-51b52562-9d6b-4524-875a-de09018ff46d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:14:17.865712 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b52562-9d6b-4524-875a-de09018ff46d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:14:17.867303 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b52562-9d6b-4524-875a-de09018ff46d service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:14:17.877147 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1242/2481] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:14:17 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:14:27.384165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:14:27.872543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=6 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:14:28.301882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a903e-1717-4783-ae84-7161a8327bed None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:14:28.322048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a903e-1717-4783-ae84-7161a8327bed None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:14:28.330364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63e1736-7c0e-48b9-9642-63711526535e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:14:28.331232 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63e1736-7c0e-48b9-9642-63711526535e service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:14:28.331659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63e1736-7c0e-48b9-9642-63711526535e service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:14:28.332034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63e1736-7c0e-48b9-9642-63711526535e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:14:28.334709 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f63e1736-7c0e-48b9-9642-63711526535e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:14:28.351371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63e1736-7c0e-48b9-9642-63711526535e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:14:28.353598 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63e1736-7c0e-48b9-9642-63711526535e service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:14:28.370444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1240/2482] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:14:28 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:14:38.362870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=6 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:15:06.356239 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63454ed3-f3e7-4746-aaea-c319870abb25 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:15:06.356911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63454ed3-f3e7-4746-aaea-c319870abb25 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:15:06.356911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63454ed3-f3e7-4746-aaea-c319870abb25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:15:06.379065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-63454ed3-f3e7-4746-aaea-c319870abb25 None None] MFA Rules not processed for user `52e5a4deed99442e8eed62f334aa670d`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:15:06.380496 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-63454ed3-f3e7-4746-aaea-c319870abb25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:15:06.447154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1243/2483] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:15:06 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:15:16.413964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a78dce-26a0-4acf-b4e8-cf482e4d43ea None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:15:16.414847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a78dce-26a0-4acf-b4e8-cf482e4d43ea None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:15:16.414847 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a78dce-26a0-4acf-b4e8-cf482e4d43ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:15:16.439303 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-e4a78dce-26a0-4acf-b4e8-cf482e4d43ea None None] MFA Rules not processed for user `56732ebe8ee24f3f9ce8f87746e8b161`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:15:16.440182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=25 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:15:16.441666 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e4a78dce-26a0-4acf-b4e8-cf482e4d43ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:15:16.451433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1241/2484] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:15:16 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:15:26.432573 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:16:09.341173 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63454ed3-f3e7-4746-aaea-c319870abb25 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:09.451278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63454ed3-f3e7-4746-aaea-c319870abb25 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:09.460208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49684a38-2103-4f26-9f52-1ac3a4a5846c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:09.461255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49684a38-2103-4f26-9f52-1ac3a4a5846c service placement] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:09.461743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49684a38-2103-4f26-9f52-1ac3a4a5846c service placement] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:09.462228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49684a38-2103-4f26-9f52-1ac3a4a5846c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:09.465307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-49684a38-2103-4f26-9f52-1ac3a4a5846c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:09.534582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49684a38-2103-4f26-9f52-1ac3a4a5846c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:09.537665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49684a38-2103-4f26-9f52-1ac3a4a5846c service placement] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:09.563898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1244/2485] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:16:09 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:16:13.178842 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cf9135-f91b-4070-aaa2-162f19f18f36 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:13.179454 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cf9135-f91b-4070-aaa2-162f19f18f36 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:13.179585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cf9135-f91b-4070-aaa2-162f19f18f36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:13.212517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-e7cf9135-f91b-4070-aaa2-162f19f18f36 None None] MFA Rules not processed for user `20df8d74c87143dd81dbaae76b3c2b62`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:16:13.213759 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e7cf9135-f91b-4070-aaa2-162f19f18f36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:13.221682 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1242/2486] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:16:13 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:16:16.850980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49684a38-2103-4f26-9f52-1ac3a4a5846c service placement] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:16.853433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-49684a38-2103-4f26-9f52-1ac3a4a5846c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:16.972309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49684a38-2103-4f26-9f52-1ac3a4a5846c service placement] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:16.976813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d10a1c2-c505-4f08-bf4d-77e6a2f8b3d3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:16.977545 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d10a1c2-c505-4f08-bf4d-77e6a2f8b3d3 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:16.977964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d10a1c2-c505-4f08-bf4d-77e6a2f8b3d3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:16.978365 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d10a1c2-c505-4f08-bf4d-77e6a2f8b3d3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:16.980842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3d10a1c2-c505-4f08-bf4d-77e6a2f8b3d3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:16.999478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d10a1c2-c505-4f08-bf4d-77e6a2f8b3d3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:17.001553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d10a1c2-c505-4f08-bf4d-77e6a2f8b3d3 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:17.018179 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1245/2487] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:16:16 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:16:23.203178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:16:27.010778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=71 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:16:42.249603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cf9135-f91b-4070-aaa2-162f19f18f36 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:42.274999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cf9135-f91b-4070-aaa2-162f19f18f36 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:42.279200 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144528bf-9b57-45b3-85a8-e291fa3fd3b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:42.280067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144528bf-9b57-45b3-85a8-e291fa3fd3b1 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:42.280549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144528bf-9b57-45b3-85a8-e291fa3fd3b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:42.281019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144528bf-9b57-45b3-85a8-e291fa3fd3b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:42.283433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-144528bf-9b57-45b3-85a8-e291fa3fd3b1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:42.408742 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144528bf-9b57-45b3-85a8-e291fa3fd3b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:42.410724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144528bf-9b57-45b3-85a8-e291fa3fd3b1 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:42.447039 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1243/2488] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:16:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:16:47.734193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d10a1c2-c505-4f08-bf4d-77e6a2f8b3d3 service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:47.738132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3d10a1c2-c505-4f08-bf4d-77e6a2f8b3d3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:47.769891 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d10a1c2-c505-4f08-bf4d-77e6a2f8b3d3 service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:47.782944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bf7fe6-bb7c-420a-b027-938f226cff24 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:47.785090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bf7fe6-bb7c-420a-b027-938f226cff24 service placement] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:47.786300 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bf7fe6-bb7c-420a-b027-938f226cff24 service placement] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:47.787125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bf7fe6-bb7c-420a-b027-938f226cff24 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:47.791627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a0bf7fe6-bb7c-420a-b027-938f226cff24 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:47.818213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bf7fe6-bb7c-420a-b027-938f226cff24 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:47.822156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bf7fe6-bb7c-420a-b027-938f226cff24 service placement] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:47.850976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1246/2489] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:16:47 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:16:47.881246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144528bf-9b57-45b3-85a8-e291fa3fd3b1 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:47.883633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-144528bf-9b57-45b3-85a8-e291fa3fd3b1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:47.913205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144528bf-9b57-45b3-85a8-e291fa3fd3b1 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:47.927200 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca188d3-caf3-4f3b-a9c1-34d06965e152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:47.928905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca188d3-caf3-4f3b-a9c1-34d06965e152 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:47.930116 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca188d3-caf3-4f3b-a9c1-34d06965e152 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:47.930943 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca188d3-caf3-4f3b-a9c1-34d06965e152 admin admin] PATH_INFO: `/v3/users/52e5a4deed99442e8eed62f334aa670d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:47.934548 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca188d3-caf3-4f3b-a9c1-34d06965e152 admin admin] RBAC: Authorizing `identity:delete_user(user_id=52e5a4deed99442e8eed62f334aa670d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:47.940159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca188d3-caf3-4f3b-a9c1-34d06965e152 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:48.040442 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-fca188d3-caf3-4f3b-a9c1-34d06965e152 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52e5a4deed99442e8eed62f334aa670d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:16:48.055802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-fca188d3-caf3-4f3b-a9c1-34d06965e152 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52e5a4deed99442e8eed62f334aa670d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:16:48.057067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-fca188d3-caf3-4f3b-a9c1-34d06965e152 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52e5a4deed99442e8eed62f334aa670d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:16:48.090115 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-fca188d3-caf3-4f3b-a9c1-34d06965e152 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52e5a4deed99442e8eed62f334aa670d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:16:48.091270 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-fca188d3-caf3-4f3b-a9c1-34d06965e152 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52e5a4deed99442e8eed62f334aa670d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:16:48.126926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1244/2490] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:16:47 2022] DELETE /identity/v3/users/52e5a4deed99442e8eed62f334aa670d => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:16:48.457705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bf7fe6-bb7c-420a-b027-938f226cff24 service placement] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:48.466390 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd82317-83c8-4481-8749-cef41e056c1e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:48.467202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd82317-83c8-4481-8749-cef41e056c1e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:48.467929 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd82317-83c8-4481-8749-cef41e056c1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:48.493757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bf7fe6-bb7c-420a-b027-938f226cff24 service placement] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:48.540848 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-fcd82317-83c8-4481-8749-cef41e056c1e None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:16:48.544415 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fcd82317-83c8-4481-8749-cef41e056c1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:48.578256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:16:48.632826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:48.633506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:48.633787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:48.634064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] PATH_INFO: `/v3/projects/3d6a76e846cb48bb91eab4da1c842f2b` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:48.635544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d6a76e846cb48bb91eab4da1c842f2b)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:48.637855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:48.696073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1245/2491] 10.222.0.42 () {58 vars in 967 bytes} [Sun Jul 31 01:16:48 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:16:48.726127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d6a76e846cb48bb91eab4da1c842f2b', 'request_id': 'req-b6ea5dc3-d476-4b4c-9f63-3118c5882775'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:16:48.737201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d6a76e846cb48bb91eab4da1c842f2b', 'request_id': 'req-b6ea5dc3-d476-4b4c-9f63-3118c5882775'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:16:48.749912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d6a76e846cb48bb91eab4da1c842f2b', 'request_id': 'req-b6ea5dc3-d476-4b4c-9f63-3118c5882775'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:16:48.775692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] The token cache is being invalidate because project 3d6a76e846cb48bb91eab4da1c842f2b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:16:48.776112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:16:48.777429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:16:48.785658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1247/2492] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:16:48 2022] DELETE /identity/v3/projects/3d6a76e846cb48bb91eab4da1c842f2b => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:16:48.790849 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f235bbff-9f4f-407e-8ecd-710808bbe67a None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:48.795609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:48.795901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f235bbff-9f4f-407e-8ecd-710808bbe67a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:48.796136 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f235bbff-9f4f-407e-8ecd-710808bbe67a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:48.798701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:48.954794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-f235bbff-9f4f-407e-8ecd-710808bbe67a None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:16:48.956894 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-f235bbff-9f4f-407e-8ecd-710808bbe67a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:49.000148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ea5dc3-d476-4b4c-9f63-3118c5882775 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:49.017966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf703c9-f99b-425f-8f2c-e78222cce07c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:49.018802 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf703c9-f99b-425f-8f2c-e78222cce07c service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:49.019150 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf703c9-f99b-425f-8f2c-e78222cce07c service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:49.019680 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf703c9-f99b-425f-8f2c-e78222cce07c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:49.023551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7bf703c9-f99b-425f-8f2c-e78222cce07c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:49.104506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:16:49.142761 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1246/2493] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:16:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 357 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:16:49.159879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fad5069-8b8c-4f73-a668-5cfcdb4239bd None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:49.160547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fad5069-8b8c-4f73-a668-5cfcdb4239bd None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:49.161323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fad5069-8b8c-4f73-a668-5cfcdb4239bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:49.205869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-4fad5069-8b8c-4f73-a668-5cfcdb4239bd None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:16:49.211195 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-4fad5069-8b8c-4f73-a668-5cfcdb4239bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:49.224000 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf703c9-f99b-425f-8f2c-e78222cce07c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:49.226101 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf703c9-f99b-425f-8f2c-e78222cce07c service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:49.229819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1247/2494] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:16:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:16:49.236077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fad5069-8b8c-4f73-a668-5cfcdb4239bd None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:49.240295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1248/2495] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:16:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 446 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 31 01:16:49.255546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fad5069-8b8c-4f73-a668-5cfcdb4239bd None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:49.261879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce0aa8c-dd4f-4503-8be6-45a1ba90b358 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:49.262630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce0aa8c-dd4f-4503-8be6-45a1ba90b358 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:49.262998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce0aa8c-dd4f-4503-8be6-45a1ba90b358 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:49.263388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce0aa8c-dd4f-4503-8be6-45a1ba90b358 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:49.264324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce0aa8c-dd4f-4503-8be6-45a1ba90b358 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:16:49.264737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce0aa8c-dd4f-4503-8be6-45a1ba90b358 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:49.266910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce0aa8c-dd4f-4503-8be6-45a1ba90b358 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:49.296024 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1248/2496] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:16:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:16:49.301287 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf703c9-f99b-425f-8f2c-e78222cce07c service swift] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:49.318045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf703c9-f99b-425f-8f2c-e78222cce07c service swift] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:49.325174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad953e40-b211-4770-98de-3c15745d9f2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:49.328826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad953e40-b211-4770-98de-3c15745d9f2c admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:49.329308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad953e40-b211-4770-98de-3c15745d9f2c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:49.329776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad953e40-b211-4770-98de-3c15745d9f2c admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:49.331147 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad953e40-b211-4770-98de-3c15745d9f2c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:49.340693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad953e40-b211-4770-98de-3c15745d9f2c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:49.375579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1249/2497] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:16:49 2022] POST /identity/v3/projects => generated 385 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:16:49.381194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce0aa8c-dd4f-4503-8be6-45a1ba90b358 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:49.397728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce0aa8c-dd4f-4503-8be6-45a1ba90b358 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:49.521705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50a08e3-9aae-45b5-a215-e09d957a8d5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:49.523590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50a08e3-9aae-45b5-a215-e09d957a8d5b admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:49.524488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50a08e3-9aae-45b5-a215-e09d957a8d5b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:49.525854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50a08e3-9aae-45b5-a215-e09d957a8d5b admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:49.527796 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50a08e3-9aae-45b5-a215-e09d957a8d5b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:49.531214 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50a08e3-9aae-45b5-a215-e09d957a8d5b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:49.565691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:16:49.571039 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1249/2498] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:16:49 2022] POST /identity/v3/users => generated 293 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:16:49.575770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad953e40-b211-4770-98de-3c15745d9f2c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:49.599201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad953e40-b211-4770-98de-3c15745d9f2c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:49.609206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1121ba94-f999-41c4-bb04-e4b37c77e89b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:49.610869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1121ba94-f999-41c4-bb04-e4b37c77e89b admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:49.611624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1121ba94-f999-41c4-bb04-e4b37c77e89b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:49.612155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1121ba94-f999-41c4-bb04-e4b37c77e89b admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:49.614371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1121ba94-f999-41c4-bb04-e4b37c77e89b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:49.617797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1121ba94-f999-41c4-bb04-e4b37c77e89b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:49.637532 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1250/2499] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:16:49 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:16:49.643029 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50a08e3-9aae-45b5-a215-e09d957a8d5b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:49.658293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50a08e3-9aae-45b5-a215-e09d957a8d5b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:49.664417 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a5463a-83f3-4da7-b5b2-6b3a13a6a1b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:49.665275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a5463a-83f3-4da7-b5b2-6b3a13a6a1b0 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:49.665803 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a5463a-83f3-4da7-b5b2-6b3a13a6a1b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:49.666103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a5463a-83f3-4da7-b5b2-6b3a13a6a1b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:49.666977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a5463a-83f3-4da7-b5b2-6b3a13a6a1b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:49.669000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a5463a-83f3-4da7-b5b2-6b3a13a6a1b0 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:49.690339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1250/2500] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:16:49 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:16:49.690583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1121ba94-f999-41c4-bb04-e4b37c77e89b admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:49.703729 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1121ba94-f999-41c4-bb04-e4b37c77e89b admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:49.709004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a687216d-3a59-4809-83f5-b189544180be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:49.709897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a687216d-3a59-4809-83f5-b189544180be admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:49.710311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a687216d-3a59-4809-83f5-b189544180be admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:49.710650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a687216d-3a59-4809-83f5-b189544180be admin admin] PATH_INFO: `/v3/projects/2dbb3f502e3f4eb591d8ca4588efa00f/users/82789ff85af6474b8db657cf4c71f6f2/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:49.735177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a687216d-3a59-4809-83f5-b189544180be admin admin] RBAC: Authorizing `identity:create_grant(project_id=2dbb3f502e3f4eb591d8ca4588efa00f, user_id=82789ff85af6474b8db657cf4c71f6f2, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:49.739068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a687216d-3a59-4809-83f5-b189544180be admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:49.777393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1251/2501] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:16:49 2022] PUT /identity/v3/projects/2dbb3f502e3f4eb591d8ca4588efa00f/users/82789ff85af6474b8db657cf4c71f6f2/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:16:49.780155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a860de9-907b-474e-bca7-c8b39abd71cd None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:49.780735 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a860de9-907b-474e-bca7-c8b39abd71cd None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:49.781188 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a860de9-907b-474e-bca7-c8b39abd71cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:49.835225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-9a860de9-907b-474e-bca7-c8b39abd71cd None None] MFA Rules not processed for user `82789ff85af6474b8db657cf4c71f6f2`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:16:49.837052 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9a860de9-907b-474e-bca7-c8b39abd71cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:49.958228 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1251/2502] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:16:49 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:16:49.979024 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a687216d-3a59-4809-83f5-b189544180be admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:49.982851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a687216d-3a59-4809-83f5-b189544180be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:50.053760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a48e2b-53fb-46e7-b02c-ea0ed48f076d None None] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:50.054259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a48e2b-53fb-46e7-b02c-ea0ed48f076d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:50.054958 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a48e2b-53fb-46e7-b02c-ea0ed48f076d None None] PATH_INFO: `/` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:50.059472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1252/2503] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:16:50 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 31 01:16:50.068086 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87153f37-fbc3-44e0-9510-026da30be9a6 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:50.068544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87153f37-fbc3-44e0-9510-026da30be9a6 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:50.069103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87153f37-fbc3-44e0-9510-026da30be9a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:50.117585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-87153f37-fbc3-44e0-9510-026da30be9a6 None None] MFA Rules not processed for user `94e868131d33472b9ded8486274a07b9`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:16:50.119074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-87153f37-fbc3-44e0-9510-026da30be9a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:50.214681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a687216d-3a59-4809-83f5-b189544180be admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:50.227866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c3530c-2089-484e-b298-f77c7b587ed0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:50.229299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c3530c-2089-484e-b298-f77c7b587ed0 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:50.230457 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c3530c-2089-484e-b298-f77c7b587ed0 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:50.231213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c3530c-2089-484e-b298-f77c7b587ed0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:50.238944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1253/2504] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:16:50 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:16:50.256266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c3530c-2089-484e-b298-f77c7b587ed0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:50.258508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c3530c-2089-484e-b298-f77c7b587ed0 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:50.273036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1252/2505] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:16:49 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:16:50.344096 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87153f37-fbc3-44e0-9510-026da30be9a6 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:50.347346 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-87153f37-fbc3-44e0-9510-026da30be9a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:50.362652 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87153f37-fbc3-44e0-9510-026da30be9a6 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:50.372070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5294d563-2899-4e31-b206-098a4410750a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:50.373039 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5294d563-2899-4e31-b206-098a4410750a service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:50.373732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5294d563-2899-4e31-b206-098a4410750a service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:50.374423 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5294d563-2899-4e31-b206-098a4410750a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:50.386259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5294d563-2899-4e31-b206-098a4410750a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:50.387876 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5294d563-2899-4e31-b206-098a4410750a service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:50.407818 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1254/2506] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:16:50 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:16:50.540244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa9df9c-e1b6-4f6f-94db-bbd10306a721 None None] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:50.541080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa9df9c-e1b6-4f6f-94db-bbd10306a721 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:50.543395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa9df9c-e1b6-4f6f-94db-bbd10306a721 None None] PATH_INFO: `/` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:50.548195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1253/2507] 10.222.0.42 () {54 vars in 921 bytes} [Sun Jul 31 01:16:50 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 31 01:16:50.560578 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c10b738-dbe5-440d-b996-f18e4fb8c651 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:50.561480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c10b738-dbe5-440d-b996-f18e4fb8c651 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:50.561868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c10b738-dbe5-440d-b996-f18e4fb8c651 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:50.593423 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-1c10b738-dbe5-440d-b996-f18e4fb8c651 None None] MFA Rules not processed for user `5b0bd572ae7a4c7f9e07fe0b9708aa7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:16:50.596682 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1c10b738-dbe5-440d-b996-f18e4fb8c651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:16:50.608246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1255/2508] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Jul 31 01:16:50 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:16:50.631317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa9df9c-e1b6-4f6f-94db-bbd10306a721 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:16:50.648436 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa9df9c-e1b6-4f6f-94db-bbd10306a721 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:16:50.658576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c572c1-1678-4cd9-b7b9-9d4dea4e1469 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:16:50.659687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c572c1-1678-4cd9-b7b9-9d4dea4e1469 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:16:50.660342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c572c1-1678-4cd9-b7b9-9d4dea4e1469 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:16:50.661294 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c572c1-1678-4cd9-b7b9-9d4dea4e1469 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:16:50.687535 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c572c1-1678-4cd9-b7b9-9d4dea4e1469 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:16:50.690334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c572c1-1678-4cd9-b7b9-9d4dea4e1469 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:16:50.730659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1254/2509] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:16:50 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:17:00.585648 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=75 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:00.710102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=92,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:00.868153 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c10b738-dbe5-440d-b996-f18e4fb8c651 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:00.997036 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c10b738-dbe5-440d-b996-f18e4fb8c651 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:01.006807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85925356-0926-4ff0-b45d-71070206c4c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:01.006999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85925356-0926-4ff0-b45d-71070206c4c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:01.007587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85925356-0926-4ff0-b45d-71070206c4c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:01.008068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85925356-0926-4ff0-b45d-71070206c4c7 admin admin] PATH_INFO: `/v3/users/82789ff85af6474b8db657cf4c71f6f2` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:01.010650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85925356-0926-4ff0-b45d-71070206c4c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=82789ff85af6474b8db657cf4c71f6f2)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:01.014747 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85925356-0926-4ff0-b45d-71070206c4c7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:01.122178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-85925356-0926-4ff0-b45d-71070206c4c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82789ff85af6474b8db657cf4c71f6f2'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:01.136673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-85925356-0926-4ff0-b45d-71070206c4c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82789ff85af6474b8db657cf4c71f6f2'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:01.138078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-85925356-0926-4ff0-b45d-71070206c4c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82789ff85af6474b8db657cf4c71f6f2'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:01.158805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-85925356-0926-4ff0-b45d-71070206c4c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82789ff85af6474b8db657cf4c71f6f2'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:01.160509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-85925356-0926-4ff0-b45d-71070206c4c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82789ff85af6474b8db657cf4c71f6f2'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:01.167483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c572c1-1678-4cd9-b7b9-9d4dea4e1469 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:01.169371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e2c572c1-1678-4cd9-b7b9-9d4dea4e1469 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:01.194181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1256/2510] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:17:00 2022] DELETE /identity/v3/users/82789ff85af6474b8db657cf4c71f6f2 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:01.203260 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8b1103-5b57-444e-9e50-4fda331b4592 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:01.203762 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8b1103-5b57-444e-9e50-4fda331b4592 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:01.204235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8b1103-5b57-444e-9e50-4fda331b4592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:01.216775 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c572c1-1678-4cd9-b7b9-9d4dea4e1469 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:01.247707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-fb8b1103-5b57-444e-9e50-4fda331b4592 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:01.250464 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fb8b1103-5b57-444e-9e50-4fda331b4592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:01.357594 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:01.358664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:01.359144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:01.359678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] PATH_INFO: `/v3/users/1776c123f98c44b69807ec2b0281b718` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:01.361900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1776c123f98c44b69807ec2b0281b718)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:01.365644 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:01.368272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1257/2511] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:01.380870 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8b1103-5b57-444e-9e50-4fda331b4592 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:01.382475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fb8b1103-5b57-444e-9e50-4fda331b4592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:01.427651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:01.465982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1776c123f98c44b69807ec2b0281b718'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:01.467162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1776c123f98c44b69807ec2b0281b718'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:01.481148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1776c123f98c44b69807ec2b0281b718'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:01.499489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8b1103-5b57-444e-9e50-4fda331b4592 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:01.504996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1776c123f98c44b69807ec2b0281b718'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:01.506002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6f62e9-0186-4185-b94f-13f83264eeac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:01.507163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6f62e9-0186-4185-b94f-13f83264eeac service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:01.508015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6f62e9-0186-4185-b94f-13f83264eeac service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:01.508509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6f62e9-0186-4185-b94f-13f83264eeac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:01.516758 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1776c123f98c44b69807ec2b0281b718'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:01.524746 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1255/2512] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:17:01 2022] DELETE /identity/v3/users/1776c123f98c44b69807ec2b0281b718 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:01.529540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6f62e9-0186-4185-b94f-13f83264eeac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:01.532263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6f62e9-0186-4185-b94f-13f83264eeac service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:01.547778 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1258/2513] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:17:01.831799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:01.833444 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:02.051930 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6f62e9-0186-4185-b94f-13f83264eeac service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:02.054903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ab6f62e9-0186-4185-b94f-13f83264eeac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:02.078920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7f552d-e120-4002-a4bd-d741d84cdd5a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:02.087696 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:02.088929 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:02.089446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:02.089963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] PATH_INFO: `/v3/projects/b018ba41816049feaa75e10489ec5281` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:02.092318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b018ba41816049feaa75e10489ec5281)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:02.096089 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:02.096531 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6f62e9-0186-4185-b94f-13f83264eeac service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:02.107817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d1aeca-d55d-43ca-b11b-ed93d1006776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:02.108553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d1aeca-d55d-43ca-b11b-ed93d1006776 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:02.108857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d1aeca-d55d-43ca-b11b-ed93d1006776 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:02.109340 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d1aeca-d55d-43ca-b11b-ed93d1006776 admin admin] PATH_INFO: `/v3/projects/2dbb3f502e3f4eb591d8ca4588efa00f` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:02.111138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d1aeca-d55d-43ca-b11b-ed93d1006776 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2dbb3f502e3f4eb591d8ca4588efa00f)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:02.113560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d1aeca-d55d-43ca-b11b-ed93d1006776 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:02.131264 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:02.172180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b018ba41816049feaa75e10489ec5281', 'request_id': 'req-7ab4a8d6-931c-49cd-811d-338c3335a0c6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.184925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b018ba41816049feaa75e10489ec5281', 'request_id': 'req-7ab4a8d6-931c-49cd-811d-338c3335a0c6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.188146 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-74d1aeca-d55d-43ca-b11b-ed93d1006776 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2dbb3f502e3f4eb591d8ca4588efa00f', 'request_id': 'req-74d1aeca-d55d-43ca-b11b-ed93d1006776'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.199284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b018ba41816049feaa75e10489ec5281', 'request_id': 'req-7ab4a8d6-931c-49cd-811d-338c3335a0c6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.199630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-74d1aeca-d55d-43ca-b11b-ed93d1006776 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2dbb3f502e3f4eb591d8ca4588efa00f', 'request_id': 'req-74d1aeca-d55d-43ca-b11b-ed93d1006776'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.207744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-74d1aeca-d55d-43ca-b11b-ed93d1006776 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2dbb3f502e3f4eb591d8ca4588efa00f', 'request_id': 'req-74d1aeca-d55d-43ca-b11b-ed93d1006776'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.216915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] The token cache is being invalidate because project b018ba41816049feaa75e10489ec5281 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:17:02.217352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.218180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.220551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1256/2514] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:17:01 2022] DELETE /identity/v3/projects/b018ba41816049feaa75e10489ec5281 => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:17:02.225027 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:02.225645 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-74d1aeca-d55d-43ca-b11b-ed93d1006776 admin admin] The token cache is being invalidate because project 2dbb3f502e3f4eb591d8ca4588efa00f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:17:02.226056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-74d1aeca-d55d-43ca-b11b-ed93d1006776 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.226916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-74d1aeca-d55d-43ca-b11b-ed93d1006776 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.227096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:02.229508 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1259/2515] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:17:02 2022] DELETE /identity/v3/projects/2dbb3f502e3f4eb591d8ca4588efa00f => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:17:02.243092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709002a6-ee25-47ab-b0c4-933b66dea771 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:02.243477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709002a6-ee25-47ab-b0c4-933b66dea771 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:02.243893 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709002a6-ee25-47ab-b0c4-933b66dea771 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:02.269579 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-709002a6-ee25-47ab-b0c4-933b66dea771 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:02.271781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-709002a6-ee25-47ab-b0c4-933b66dea771 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:02.332362 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab4a8d6-931c-49cd-811d-338c3335a0c6 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:02.367641 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1260/2516] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:17:02.378966 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646c15fc-27fc-43f7-8b75-5804669f71c2 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:02.379330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646c15fc-27fc-43f7-8b75-5804669f71c2 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:02.379571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646c15fc-27fc-43f7-8b75-5804669f71c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:02.413345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-646c15fc-27fc-43f7-8b75-5804669f71c2 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:02.415323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-646c15fc-27fc-43f7-8b75-5804669f71c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:02.430121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1261/2517] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:17:02.435445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646c15fc-27fc-43f7-8b75-5804669f71c2 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:02.439957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:02.440766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:02.442105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:02.442105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] PATH_INFO: `/v3/users/025b941d2db04cd2b4fc018ed0fdf5b2` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:02.445910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] RBAC: Authorizing `identity:delete_user(user_id=025b941d2db04cd2b4fc018ed0fdf5b2)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:02.448500 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646c15fc-27fc-43f7-8b75-5804669f71c2 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:02.448960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:02.455053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64c85dc-1c0b-4835-adde-5f47e86a4724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:02.455850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64c85dc-1c0b-4835-adde-5f47e86a4724 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:02.456137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64c85dc-1c0b-4835-adde-5f47e86a4724 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:02.456413 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64c85dc-1c0b-4835-adde-5f47e86a4724 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:02.457131 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64c85dc-1c0b-4835-adde-5f47e86a4724 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:17:02.457438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64c85dc-1c0b-4835-adde-5f47e86a4724 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:02.459258 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64c85dc-1c0b-4835-adde-5f47e86a4724 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:02.484568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1262/2518] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:17:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:17:02.488732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64c85dc-1c0b-4835-adde-5f47e86a4724 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:02.505955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64c85dc-1c0b-4835-adde-5f47e86a4724 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:02.514949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99e930d-b849-4b84-8431-e0d9a2e1c5fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:02.515210 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99e930d-b849-4b84-8431-e0d9a2e1c5fa admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:02.515594 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99e930d-b849-4b84-8431-e0d9a2e1c5fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:02.516293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99e930d-b849-4b84-8431-e0d9a2e1c5fa admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:02.517514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99e930d-b849-4b84-8431-e0d9a2e1c5fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:02.529364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99e930d-b849-4b84-8431-e0d9a2e1c5fa admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:02.547809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '025b941d2db04cd2b4fc018ed0fdf5b2'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.548874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '025b941d2db04cd2b4fc018ed0fdf5b2'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.568240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '025b941d2db04cd2b4fc018ed0fdf5b2'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.577528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:02.604190 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '025b941d2db04cd2b4fc018ed0fdf5b2'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.604927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1263/2519] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:17:02 2022] POST /identity/v3/projects => generated 373 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:17:02.610922 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99e930d-b849-4b84-8431-e0d9a2e1c5fa admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:02.627780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:02.642102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '025b941d2db04cd2b4fc018ed0fdf5b2'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:02.655766 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1257/2520] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:17:02 2022] DELETE /identity/v3/users/025b941d2db04cd2b4fc018ed0fdf5b2 => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:17:02.670164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:02.672769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:02.758412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99e930d-b849-4b84-8431-e0d9a2e1c5fa admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:02.758963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efe4646-c349-44de-8e24-3648ce089c07 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:02.764657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b14d517-3156-4b95-a65d-3f878cb5f27f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:02.765303 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b14d517-3156-4b95-a65d-3f878cb5f27f service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:02.765668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b14d517-3156-4b95-a65d-3f878cb5f27f service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:02.766488 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b14d517-3156-4b95-a65d-3f878cb5f27f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:02.773736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0b14d517-3156-4b95-a65d-3f878cb5f27f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:02.852238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27cc5e9-15f3-404f-a73b-261bfef48c6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:02.853051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27cc5e9-15f3-404f-a73b-261bfef48c6c admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:02.853292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27cc5e9-15f3-404f-a73b-261bfef48c6c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:02.853675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27cc5e9-15f3-404f-a73b-261bfef48c6c admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:02.854297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27cc5e9-15f3-404f-a73b-261bfef48c6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:02.856428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27cc5e9-15f3-404f-a73b-261bfef48c6c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:02.881859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1264/2521] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:17:02 2022] POST /identity/v3/users => generated 287 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:17:02.883869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b14d517-3156-4b95-a65d-3f878cb5f27f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:02.888125 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27cc5e9-15f3-404f-a73b-261bfef48c6c admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:02.888760 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b14d517-3156-4b95-a65d-3f878cb5f27f service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:02.890904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a27cc5e9-15f3-404f-a73b-261bfef48c6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:02.904509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1258/2522] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:17:02.912313 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27cc5e9-15f3-404f-a73b-261bfef48c6c admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:02.920772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad88b1a9-d850-4990-a3ca-a43725387713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:02.921868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad88b1a9-d850-4990-a3ca-a43725387713 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:02.922376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad88b1a9-d850-4990-a3ca-a43725387713 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:02.922879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad88b1a9-d850-4990-a3ca-a43725387713 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:02.924249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad88b1a9-d850-4990-a3ca-a43725387713 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:02.927892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad88b1a9-d850-4990-a3ca-a43725387713 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:02.942307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1265/2523] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:02 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:17:02.948393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b14d517-3156-4b95-a65d-3f878cb5f27f service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:02.966861 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b14d517-3156-4b95-a65d-3f878cb5f27f service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:02.974611 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e36be4-6c0d-4dc5-8548-cff5f91faa2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:02.975685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e36be4-6c0d-4dc5-8548-cff5f91faa2d admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:02.976159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e36be4-6c0d-4dc5-8548-cff5f91faa2d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:02.976651 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e36be4-6c0d-4dc5-8548-cff5f91faa2d admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:02.977939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e36be4-6c0d-4dc5-8548-cff5f91faa2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:02.981218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e36be4-6c0d-4dc5-8548-cff5f91faa2d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:03.001957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1259/2524] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:02 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:17:03.008915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad88b1a9-d850-4990-a3ca-a43725387713 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:03.025438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad88b1a9-d850-4990-a3ca-a43725387713 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:03.032790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80e0f0e-c8a5-42e8-a439-12ad5f92ca2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:03.033566 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80e0f0e-c8a5-42e8-a439-12ad5f92ca2e admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:03.033953 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80e0f0e-c8a5-42e8-a439-12ad5f92ca2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:03.034294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80e0f0e-c8a5-42e8-a439-12ad5f92ca2e admin admin] PATH_INFO: `/v3/projects/bb37e790a47c4e15a7e14fe6702ec1d5/users/952605aec2924e518736b1d11e1d9f3b/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:03.053237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80e0f0e-c8a5-42e8-a439-12ad5f92ca2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb37e790a47c4e15a7e14fe6702ec1d5, user_id=952605aec2924e518736b1d11e1d9f3b, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:03.056106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80e0f0e-c8a5-42e8-a439-12ad5f92ca2e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:03.085906 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1266/2525] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:17:03 2022] PUT /identity/v3/projects/bb37e790a47c4e15a7e14fe6702ec1d5/users/952605aec2924e518736b1d11e1d9f3b/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:03.097607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e8e000-2b34-4346-a28c-4aed0b2129d6 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:03.097876 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e8e000-2b34-4346-a28c-4aed0b2129d6 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:03.098155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e8e000-2b34-4346-a28c-4aed0b2129d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:03.132633 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f0e8e000-2b34-4346-a28c-4aed0b2129d6 None None] MFA Rules not processed for user `952605aec2924e518736b1d11e1d9f3b`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:03.133975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f0e8e000-2b34-4346-a28c-4aed0b2129d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:03.164528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80e0f0e-c8a5-42e8-a439-12ad5f92ca2e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:03.167178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a80e0f0e-c8a5-42e8-a439-12ad5f92ca2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:03.257659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1260/2526] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:03 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:03.268032 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80e0f0e-c8a5-42e8-a439-12ad5f92ca2e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:03.273362 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e8e000-2b34-4346-a28c-4aed0b2129d6 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:03.275108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f0e8e000-2b34-4346-a28c-4aed0b2129d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:03.275827 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:03.276730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:03.277197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:03.277580 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb admin admin] PATH_INFO: `/v3/projects/771efa3d0e2b43e4a6394dbd92b85be1` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:03.279082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb admin admin] RBAC: Authorizing `identity:delete_project(project_id=771efa3d0e2b43e4a6394dbd92b85be1)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:03.281327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:03.283335 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=2,keystone:SELECT=98 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:03.293402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:03.355112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '771efa3d0e2b43e4a6394dbd92b85be1', 'request_id': 'req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:03.363134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '771efa3d0e2b43e4a6394dbd92b85be1', 'request_id': 'req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:03.371119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '771efa3d0e2b43e4a6394dbd92b85be1', 'request_id': 'req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:03.387869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb admin admin] The token cache is being invalidate because project 771efa3d0e2b43e4a6394dbd92b85be1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:17:03.388175 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:03.388851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e8e000-2b34-4346-a28c-4aed0b2129d6 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:03.389488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ff25e1b9-c38c-4e5f-93a0-bef373b6d1fb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:03.392615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1267/2527] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:17:03 2022] DELETE /identity/v3/projects/771efa3d0e2b43e4a6394dbd92b85be1 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:03.400214 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf37908-7960-488b-936a-1a1d8eb11830 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:03.400668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf37908-7960-488b-936a-1a1d8eb11830 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:03.401117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf37908-7960-488b-936a-1a1d8eb11830 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:03.433277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-fbf37908-7960-488b-936a-1a1d8eb11830 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:03.435829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fbf37908-7960-488b-936a-1a1d8eb11830 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:03.487249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487f07ed-0cba-49eb-b2a8-52b26d5dd731 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:03.487994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487f07ed-0cba-49eb-b2a8-52b26d5dd731 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:03.488331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487f07ed-0cba-49eb-b2a8-52b26d5dd731 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:03.488764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487f07ed-0cba-49eb-b2a8-52b26d5dd731 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:03.491168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-487f07ed-0cba-49eb-b2a8-52b26d5dd731 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:03.519571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1268/2528] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:17:03.530134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd22236e-b2b3-4424-960a-36dc2709419b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:03.530576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd22236e-b2b3-4424-960a-36dc2709419b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:03.531187 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd22236e-b2b3-4424-960a-36dc2709419b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:03.557767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-fd22236e-b2b3-4424-960a-36dc2709419b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:03.559322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fd22236e-b2b3-4424-960a-36dc2709419b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:03.570329 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1269/2529] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:17:03.575685 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd22236e-b2b3-4424-960a-36dc2709419b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:03.579326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487f07ed-0cba-49eb-b2a8-52b26d5dd731 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:03.581400 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487f07ed-0cba-49eb-b2a8-52b26d5dd731 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:03.589779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd22236e-b2b3-4424-960a-36dc2709419b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:03.594806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1261/2530] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:03 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:17:03.597492 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181ead8a-9e08-40cb-92cc-268def92df18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:03.598031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ead8a-9e08-40cb-92cc-268def92df18 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:03.598324 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ead8a-9e08-40cb-92cc-268def92df18 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:03.598651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ead8a-9e08-40cb-92cc-268def92df18 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:03.599453 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181ead8a-9e08-40cb-92cc-268def92df18 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:17:03.599798 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181ead8a-9e08-40cb-92cc-268def92df18 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:03.601556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181ead8a-9e08-40cb-92cc-268def92df18 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:03.618829 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1270/2531] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:17:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:17:03.626000 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487f07ed-0cba-49eb-b2a8-52b26d5dd731 service nova] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:03.642406 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487f07ed-0cba-49eb-b2a8-52b26d5dd731 service nova] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:03.652336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beca487d-d2f2-49d5-9c88-15f248d71804 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:03.653437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beca487d-d2f2-49d5-9c88-15f248d71804 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:03.653950 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beca487d-d2f2-49d5-9c88-15f248d71804 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:03.654681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beca487d-d2f2-49d5-9c88-15f248d71804 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:03.655956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beca487d-d2f2-49d5-9c88-15f248d71804 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:03.659890 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beca487d-d2f2-49d5-9c88-15f248d71804 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:03.714886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1262/2532] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:17:03 2022] POST /identity/v3/projects => generated 379 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:17:03.720517 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181ead8a-9e08-40cb-92cc-268def92df18 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:03.736940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181ead8a-9e08-40cb-92cc-268def92df18 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:03.842857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19253cef-bce3-4062-a0c0-a8170b32624c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:03.844082 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19253cef-bce3-4062-a0c0-a8170b32624c admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:03.844804 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19253cef-bce3-4062-a0c0-a8170b32624c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:03.845337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19253cef-bce3-4062-a0c0-a8170b32624c admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:03.848209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19253cef-bce3-4062-a0c0-a8170b32624c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:03.852000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19253cef-bce3-4062-a0c0-a8170b32624c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:03.890162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1271/2533] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:17:03 2022] POST /identity/v3/users => generated 290 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:17:03.897340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beca487d-d2f2-49d5-9c88-15f248d71804 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:03.936476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beca487d-d2f2-49d5-9c88-15f248d71804 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:03.943029 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cd779a-14e5-43bd-a87d-72f032f7696c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:03.943830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cd779a-14e5-43bd-a87d-72f032f7696c admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:03.944441 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cd779a-14e5-43bd-a87d-72f032f7696c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:03.944850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cd779a-14e5-43bd-a87d-72f032f7696c admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:03.945948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cd779a-14e5-43bd-a87d-72f032f7696c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:03.949056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cd779a-14e5-43bd-a87d-72f032f7696c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:03.971607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1263/2534] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:03 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:17:03.975855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19253cef-bce3-4062-a0c0-a8170b32624c admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:03.997013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19253cef-bce3-4062-a0c0-a8170b32624c admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:04.005213 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4533c44c-0ffc-4a69-a570-ea07586bfcbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:04.006217 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4533c44c-0ffc-4a69-a570-ea07586bfcbf admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:04.006874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4533c44c-0ffc-4a69-a570-ea07586bfcbf admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:04.007328 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4533c44c-0ffc-4a69-a570-ea07586bfcbf admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:04.008575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4533c44c-0ffc-4a69-a570-ea07586bfcbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:04.012267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4533c44c-0ffc-4a69-a570-ea07586bfcbf admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:04.015153 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:04.040679 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1272/2535] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:03 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:17:04.047710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cd779a-14e5-43bd-a87d-72f032f7696c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:04.069154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cd779a-14e5-43bd-a87d-72f032f7696c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:04.081745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c42cc6-8a3d-44c0-842d-ed6070cfc91d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:04.083083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c42cc6-8a3d-44c0-842d-ed6070cfc91d admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:04.083703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c42cc6-8a3d-44c0-842d-ed6070cfc91d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:04.084416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c42cc6-8a3d-44c0-842d-ed6070cfc91d admin admin] PATH_INFO: `/v3/projects/145c3d8e13bd4e2793bc930d8638eac1/users/6a00c618dd284f159d88ad41c0390c5e/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:04.088856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:04.116964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c42cc6-8a3d-44c0-842d-ed6070cfc91d admin admin] RBAC: Authorizing `identity:create_grant(project_id=145c3d8e13bd4e2793bc930d8638eac1, user_id=6a00c618dd284f159d88ad41c0390c5e, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:04.121804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c42cc6-8a3d-44c0-842d-ed6070cfc91d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:04.149405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1264/2536] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:17:04 2022] PUT /identity/v3/projects/145c3d8e13bd4e2793bc930d8638eac1/users/6a00c618dd284f159d88ad41c0390c5e/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:04.167557 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9cf86b-d3df-41b0-a6e9-e0355038baf7 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:04.168374 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9cf86b-d3df-41b0-a6e9-e0355038baf7 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:04.168563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9cf86b-d3df-41b0-a6e9-e0355038baf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:04.203011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-6b9cf86b-d3df-41b0-a6e9-e0355038baf7 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:04.205051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6b9cf86b-d3df-41b0-a6e9-e0355038baf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:04.313337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1273/2537] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:04.323973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c42cc6-8a3d-44c0-842d-ed6070cfc91d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:04.325790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-92c42cc6-8a3d-44c0-842d-ed6070cfc91d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:04.396978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c42cc6-8a3d-44c0-842d-ed6070cfc91d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:04.400786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12780aef-0212-4725-b38f-3c3dda779b10 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:04.401439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12780aef-0212-4725-b38f-3c3dda779b10 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:04.401714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12780aef-0212-4725-b38f-3c3dda779b10 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:04.401988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12780aef-0212-4725-b38f-3c3dda779b10 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:04.412097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12780aef-0212-4725-b38f-3c3dda779b10 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:04.413822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12780aef-0212-4725-b38f-3c3dda779b10 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:04.425823 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1265/2538] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:17:06.901657 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9cf86b-d3df-41b0-a6e9-e0355038baf7 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:06.903377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6b9cf86b-d3df-41b0-a6e9-e0355038baf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:06.917264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9cf86b-d3df-41b0-a6e9-e0355038baf7 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:06.923912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:06.925440 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:06.925836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:06.926220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] PATH_INFO: `/v3/users/952605aec2924e518736b1d11e1d9f3b` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:06.928694 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] RBAC: Authorizing `identity:delete_user(user_id=952605aec2924e518736b1d11e1d9f3b)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:06.930838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:06.997504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '952605aec2924e518736b1d11e1d9f3b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:07.009920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '952605aec2924e518736b1d11e1d9f3b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:07.010734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '952605aec2924e518736b1d11e1d9f3b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:07.030037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '952605aec2924e518736b1d11e1d9f3b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:07.031078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '952605aec2924e518736b1d11e1d9f3b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:07.057771 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1274/2539] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:17:06 2022] DELETE /identity/v3/users/952605aec2924e518736b1d11e1d9f3b => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:17:07.061168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32013516-684d-4aca-9359-97e9a6068f0f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:07.063036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32013516-684d-4aca-9359-97e9a6068f0f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:07.063783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32013516-684d-4aca-9359-97e9a6068f0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:07.105068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-32013516-684d-4aca-9359-97e9a6068f0f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:07.107206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-32013516-684d-4aca-9359-97e9a6068f0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:07.220450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1266/2540] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:17:07.233632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:07.242704 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:07.380988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c3eb59-7981-4b43-a847-8e6275faec1e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:07.388983 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754c1fb5-668b-4256-bd97-99b26787e35d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:07.390189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754c1fb5-668b-4256-bd97-99b26787e35d service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:07.392513 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754c1fb5-668b-4256-bd97-99b26787e35d service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:07.393606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754c1fb5-668b-4256-bd97-99b26787e35d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:07.408376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754c1fb5-668b-4256-bd97-99b26787e35d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:07.410198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754c1fb5-668b-4256-bd97-99b26787e35d service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:07.426170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1275/2541] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:17:08.045302 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32013516-684d-4aca-9359-97e9a6068f0f None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:08.048674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-32013516-684d-4aca-9359-97e9a6068f0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:08.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32013516-684d-4aca-9359-97e9a6068f0f None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:08.128636 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f92848-e141-45b5-8267-8a24a41d69d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:08.130620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f92848-e141-45b5-8267-8a24a41d69d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:08.135751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f92848-e141-45b5-8267-8a24a41d69d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:08.137503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f92848-e141-45b5-8267-8a24a41d69d6 admin admin] PATH_INFO: `/v3/projects/bb37e790a47c4e15a7e14fe6702ec1d5` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:08.140152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f92848-e141-45b5-8267-8a24a41d69d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb37e790a47c4e15a7e14fe6702ec1d5)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:08.145506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f92848-e141-45b5-8267-8a24a41d69d6 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:08.312128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-22f92848-e141-45b5-8267-8a24a41d69d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb37e790a47c4e15a7e14fe6702ec1d5', 'request_id': 'req-22f92848-e141-45b5-8267-8a24a41d69d6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:08.338149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-22f92848-e141-45b5-8267-8a24a41d69d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb37e790a47c4e15a7e14fe6702ec1d5', 'request_id': 'req-22f92848-e141-45b5-8267-8a24a41d69d6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:08.359083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-22f92848-e141-45b5-8267-8a24a41d69d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb37e790a47c4e15a7e14fe6702ec1d5', 'request_id': 'req-22f92848-e141-45b5-8267-8a24a41d69d6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:08.406958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-22f92848-e141-45b5-8267-8a24a41d69d6 admin admin] The token cache is being invalidate because project bb37e790a47c4e15a7e14fe6702ec1d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:17:08.407741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-22f92848-e141-45b5-8267-8a24a41d69d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:08.409334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-22f92848-e141-45b5-8267-8a24a41d69d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:08.413019 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1267/2542] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:17:08 2022] DELETE /identity/v3/projects/bb37e790a47c4e15a7e14fe6702ec1d5 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:08.418301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d50a45-6876-4bd7-9ddf-2e5afc33959b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:08.418989 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d50a45-6876-4bd7-9ddf-2e5afc33959b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:08.418989 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d50a45-6876-4bd7-9ddf-2e5afc33959b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:08.459763 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-75d50a45-6876-4bd7-9ddf-2e5afc33959b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:08.461581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-75d50a45-6876-4bd7-9ddf-2e5afc33959b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:08.477106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:08.631690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1276/2543] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:08.641230 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f63842-df6c-453a-8011-f1d815ca94bb None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:08.641836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f63842-df6c-453a-8011-f1d815ca94bb None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:08.642527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f63842-df6c-453a-8011-f1d815ca94bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:08.669582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f2f63842-df6c-453a-8011-f1d815ca94bb None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:08.670957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f2f63842-df6c-453a-8011-f1d815ca94bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:08.683159 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1268/2544] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:08.692655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d50a45-6876-4bd7-9ddf-2e5afc33959b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:08.711555 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d50a45-6876-4bd7-9ddf-2e5afc33959b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:08.716980 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84e7ce7-c7f7-43f1-8c1f-9f0f19b2964b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:08.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e7ce7-c7f7-43f1-8c1f-9f0f19b2964b admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:08.717908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e7ce7-c7f7-43f1-8c1f-9f0f19b2964b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:08.718176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e7ce7-c7f7-43f1-8c1f-9f0f19b2964b admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:08.718904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84e7ce7-c7f7-43f1-8c1f-9f0f19b2964b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:17:08.719203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84e7ce7-c7f7-43f1-8c1f-9f0f19b2964b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:08.720921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84e7ce7-c7f7-43f1-8c1f-9f0f19b2964b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:08.736994 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1277/2545] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:17:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:17:08.741955 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f63842-df6c-453a-8011-f1d815ca94bb None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:08.748230 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:08.761852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f63842-df6c-453a-8011-f1d815ca94bb None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:08.772910 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c96af0-3c15-444b-8127-bc6a3fdcd866 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:08.775658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c96af0-3c15-444b-8127-bc6a3fdcd866 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:08.775658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c96af0-3c15-444b-8127-bc6a3fdcd866 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:08.775658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c96af0-3c15-444b-8127-bc6a3fdcd866 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:08.776224 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c96af0-3c15-444b-8127-bc6a3fdcd866 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:08.779716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c96af0-3c15-444b-8127-bc6a3fdcd866 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:08.834218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1269/2546] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:17:08 2022] POST /identity/v3/projects => generated 363 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:17:08.839249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84e7ce7-c7f7-43f1-8c1f-9f0f19b2964b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:08.880481 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84e7ce7-c7f7-43f1-8c1f-9f0f19b2964b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:08.982745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae06d38-975b-4d15-bc06-cec79f1202a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:08.982745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae06d38-975b-4d15-bc06-cec79f1202a4 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:08.982745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae06d38-975b-4d15-bc06-cec79f1202a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:08.983574 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae06d38-975b-4d15-bc06-cec79f1202a4 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:08.984467 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae06d38-975b-4d15-bc06-cec79f1202a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:08.986525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae06d38-975b-4d15-bc06-cec79f1202a4 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:09.020559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1278/2547] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:17:08 2022] POST /identity/v3/users => generated 282 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:17:09.025211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c96af0-3c15-444b-8127-bc6a3fdcd866 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:09.067837 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c96af0-3c15-444b-8127-bc6a3fdcd866 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:09.077947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62affcee-fae5-4edb-8f25-6a7b03a07733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:09.079127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62affcee-fae5-4edb-8f25-6a7b03a07733 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:09.079672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62affcee-fae5-4edb-8f25-6a7b03a07733 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:09.080174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62affcee-fae5-4edb-8f25-6a7b03a07733 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:09.081566 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62affcee-fae5-4edb-8f25-6a7b03a07733 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:09.085133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62affcee-fae5-4edb-8f25-6a7b03a07733 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:09.108034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1270/2548] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:09 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:17:09.112283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae06d38-975b-4d15-bc06-cec79f1202a4 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:09.130560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae06d38-975b-4d15-bc06-cec79f1202a4 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:09.140420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9ad66b-1d14-4872-a702-72aec4d6bc57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:09.141725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9ad66b-1d14-4872-a702-72aec4d6bc57 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:09.142491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9ad66b-1d14-4872-a702-72aec4d6bc57 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:09.143643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9ad66b-1d14-4872-a702-72aec4d6bc57 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:09.144569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9ad66b-1d14-4872-a702-72aec4d6bc57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:09.147864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9ad66b-1d14-4872-a702-72aec4d6bc57 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:09.170091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1279/2549] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:09 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:17:09.175110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62affcee-fae5-4edb-8f25-6a7b03a07733 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:09.201595 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62affcee-fae5-4edb-8f25-6a7b03a07733 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:09.211303 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6da6a8d-e385-4b2a-abfc-c7ec05e0b368 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:09.213941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6da6a8d-e385-4b2a-abfc-c7ec05e0b368 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:09.213941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6da6a8d-e385-4b2a-abfc-c7ec05e0b368 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:09.214293 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6da6a8d-e385-4b2a-abfc-c7ec05e0b368 admin admin] PATH_INFO: `/v3/projects/35a46b7fdfe24f2d85cf7a738da77c18/users/625410c1d24d47668b886409148a94af/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:09.245077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6da6a8d-e385-4b2a-abfc-c7ec05e0b368 admin admin] RBAC: Authorizing `identity:create_grant(project_id=35a46b7fdfe24f2d85cf7a738da77c18, user_id=625410c1d24d47668b886409148a94af, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:09.250517 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6da6a8d-e385-4b2a-abfc-c7ec05e0b368 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:09.299404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1271/2550] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:17:09 2022] PUT /identity/v3/projects/35a46b7fdfe24f2d85cf7a738da77c18/users/625410c1d24d47668b886409148a94af/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:09.311913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fd6662-1376-4adc-880b-29968ed3713e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:09.312505 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fd6662-1376-4adc-880b-29968ed3713e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:09.313493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fd6662-1376-4adc-880b-29968ed3713e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:09.369043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a1fd6662-1376-4adc-880b-29968ed3713e None None] MFA Rules not processed for user `625410c1d24d47668b886409148a94af`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:09.371480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a1fd6662-1376-4adc-880b-29968ed3713e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:09.536081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1280/2551] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:09 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:09.537998 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6da6a8d-e385-4b2a-abfc-c7ec05e0b368 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:09.701480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6da6a8d-e385-4b2a-abfc-c7ec05e0b368 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:09.709687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400f661e-e03f-493b-8fc2-8c204c3cf789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:09.710355 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400f661e-e03f-493b-8fc2-8c204c3cf789 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:09.710649 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400f661e-e03f-493b-8fc2-8c204c3cf789 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:09.710955 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400f661e-e03f-493b-8fc2-8c204c3cf789 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:09.712523 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400f661e-e03f-493b-8fc2-8c204c3cf789 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:09.713748 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400f661e-e03f-493b-8fc2-8c204c3cf789 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:09.776155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1272/2552] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:17:09 2022] POST /identity/v3/projects => generated 365 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:17:09.780454 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fd6662-1376-4adc-880b-29968ed3713e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:09.795684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fd6662-1376-4adc-880b-29968ed3713e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:09.809449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:09.888379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cf7253-5860-4f22-8d0a-c016b9d3bd3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:09.889094 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cf7253-5860-4f22-8d0a-c016b9d3bd3a admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:09.889427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cf7253-5860-4f22-8d0a-c016b9d3bd3a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:09.889728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cf7253-5860-4f22-8d0a-c016b9d3bd3a admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:09.890466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cf7253-5860-4f22-8d0a-c016b9d3bd3a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:09.893364 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cf7253-5860-4f22-8d0a-c016b9d3bd3a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:09.935436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1281/2553] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:17:09 2022] POST /identity/v3/users => generated 289 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:17:09.935975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400f661e-e03f-493b-8fc2-8c204c3cf789 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:09.947871 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400f661e-e03f-493b-8fc2-8c204c3cf789 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:09.953879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf66ea14-3c89-42e9-b934-f32fe748a48d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:09.954512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf66ea14-3c89-42e9-b934-f32fe748a48d admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:09.954792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf66ea14-3c89-42e9-b934-f32fe748a48d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:09.955051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf66ea14-3c89-42e9-b934-f32fe748a48d admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:09.955993 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf66ea14-3c89-42e9-b934-f32fe748a48d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:09.959103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf66ea14-3c89-42e9-b934-f32fe748a48d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:09.971429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1273/2554] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:09 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:17:09.976011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cf7253-5860-4f22-8d0a-c016b9d3bd3a admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:09.993887 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cf7253-5860-4f22-8d0a-c016b9d3bd3a admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:10.001381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc360ee3-4711-41fd-b6e2-8773539c6879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:10.002682 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc360ee3-4711-41fd-b6e2-8773539c6879 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:10.002682 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc360ee3-4711-41fd-b6e2-8773539c6879 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:10.003095 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc360ee3-4711-41fd-b6e2-8773539c6879 admin admin] PATH_INFO: `/v3/projects/c1ce1171c40e4d9099b2ab4e334b9738/users/098d180d26e54458935c84063e4c416d/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:10.024518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc360ee3-4711-41fd-b6e2-8773539c6879 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1ce1171c40e4d9099b2ab4e334b9738, user_id=098d180d26e54458935c84063e4c416d, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:10.029298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc360ee3-4711-41fd-b6e2-8773539c6879 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:10.050615 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1282/2555] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:17:09 2022] PUT /identity/v3/projects/c1ce1171c40e4d9099b2ab4e334b9738/users/098d180d26e54458935c84063e4c416d/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:17:10.055848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf66ea14-3c89-42e9-b934-f32fe748a48d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:10.068529 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf66ea14-3c89-42e9-b934-f32fe748a48d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:10.185483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4514abcf-b31f-405c-9606-0fbd33202eca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:10.187314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4514abcf-b31f-405c-9606-0fbd33202eca admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:10.188238 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4514abcf-b31f-405c-9606-0fbd33202eca admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:10.188946 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4514abcf-b31f-405c-9606-0fbd33202eca admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:10.190992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4514abcf-b31f-405c-9606-0fbd33202eca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:10.196536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4514abcf-b31f-405c-9606-0fbd33202eca admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:10.219127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1274/2556] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:10 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:17:10.221518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc360ee3-4711-41fd-b6e2-8773539c6879 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:10.245396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc360ee3-4711-41fd-b6e2-8773539c6879 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:10.254813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fb6ef5-d194-4497-b374-c501b9bd067a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:10.256394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fb6ef5-d194-4497-b374-c501b9bd067a admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:10.257186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fb6ef5-d194-4497-b374-c501b9bd067a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:10.257977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fb6ef5-d194-4497-b374-c501b9bd067a admin admin] PATH_INFO: `/v3/projects/c1ce1171c40e4d9099b2ab4e334b9738/users/098d180d26e54458935c84063e4c416d/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:10.262425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fb6ef5-d194-4497-b374-c501b9bd067a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1ce1171c40e4d9099b2ab4e334b9738, user_id=098d180d26e54458935c84063e4c416d, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:10.267419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fb6ef5-d194-4497-b374-c501b9bd067a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:10.292614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1283/2557] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:17:10 2022] PUT /identity/v3/projects/c1ce1171c40e4d9099b2ab4e334b9738/users/098d180d26e54458935c84063e4c416d/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:10.314366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afb9f95-7020-4f93-b57f-7ad5e7434f99 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:10.314863 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afb9f95-7020-4f93-b57f-7ad5e7434f99 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:10.315348 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afb9f95-7020-4f93-b57f-7ad5e7434f99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:10.388480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-4afb9f95-7020-4f93-b57f-7ad5e7434f99 None None] MFA Rules not processed for user `098d180d26e54458935c84063e4c416d`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:10.390015 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4afb9f95-7020-4f93-b57f-7ad5e7434f99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:10.394945 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:10.574916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1275/2558] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:10 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:10.590018 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fb6ef5-d194-4497-b374-c501b9bd067a admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:10.593343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-58fb6ef5-d194-4497-b374-c501b9bd067a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:10.761102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fb6ef5-d194-4497-b374-c501b9bd067a admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:10.770445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7a2731-ab4f-4385-8f7c-b6ea55ba8c08 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:10.771868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7a2731-ab4f-4385-8f7c-b6ea55ba8c08 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:10.772379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7a2731-ab4f-4385-8f7c-b6ea55ba8c08 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:10.773471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7a2731-ab4f-4385-8f7c-b6ea55ba8c08 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:10.796894 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7a2731-ab4f-4385-8f7c-b6ea55ba8c08 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:10.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7a2731-ab4f-4385-8f7c-b6ea55ba8c08 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:10.829868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1284/2559] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:10 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:17:11.650451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afb9f95-7020-4f93-b57f-7ad5e7434f99 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:11.789110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afb9f95-7020-4f93-b57f-7ad5e7434f99 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:11.800534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33184fd3-e8cf-441c-a116-ed9553988f38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:11.802057 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33184fd3-e8cf-441c-a116-ed9553988f38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:11.802339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33184fd3-e8cf-441c-a116-ed9553988f38 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:11.802895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33184fd3-e8cf-441c-a116-ed9553988f38 admin admin] PATH_INFO: `/v3/users/625410c1d24d47668b886409148a94af` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:11.804653 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33184fd3-e8cf-441c-a116-ed9553988f38 admin admin] RBAC: Authorizing `identity:delete_user(user_id=625410c1d24d47668b886409148a94af)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:11.807994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33184fd3-e8cf-441c-a116-ed9553988f38 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:11.829579 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585d200a-641c-40de-af11-b25def3ae4fb None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:11.830996 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585d200a-641c-40de-af11-b25def3ae4fb None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:11.831515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585d200a-641c-40de-af11-b25def3ae4fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:11.881607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-585d200a-641c-40de-af11-b25def3ae4fb None None] MFA Rules not processed for user `6a00c618dd284f159d88ad41c0390c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:11.883667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-585d200a-641c-40de-af11-b25def3ae4fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:11.926603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-33184fd3-e8cf-441c-a116-ed9553988f38 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '625410c1d24d47668b886409148a94af'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:11.927393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-33184fd3-e8cf-441c-a116-ed9553988f38 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '625410c1d24d47668b886409148a94af'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:11.949116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-33184fd3-e8cf-441c-a116-ed9553988f38 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '625410c1d24d47668b886409148a94af'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:11.982703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-33184fd3-e8cf-441c-a116-ed9553988f38 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '625410c1d24d47668b886409148a94af'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:11.997586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-33184fd3-e8cf-441c-a116-ed9553988f38 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '625410c1d24d47668b886409148a94af'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:12.021353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1276/2560] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:17:11 2022] DELETE /identity/v3/users/625410c1d24d47668b886409148a94af => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:12.028769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255d2c8a-f552-4123-ba3d-a1f068ed7586 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:12.029463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255d2c8a-f552-4123-ba3d-a1f068ed7586 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:12.029782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255d2c8a-f552-4123-ba3d-a1f068ed7586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:12.038236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:12.077120 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-255d2c8a-f552-4123-ba3d-a1f068ed7586 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:12.081683 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-255d2c8a-f552-4123-ba3d-a1f068ed7586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:12.103891 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:12.117279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1285/2561] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:11 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:12.126483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585d200a-641c-40de-af11-b25def3ae4fb None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:12.129547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-585d200a-641c-40de-af11-b25def3ae4fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:12.161156 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585d200a-641c-40de-af11-b25def3ae4fb None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:12.257546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee2eb29-d2c9-4562-a8e4-8e3c82a54c1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:12.258494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2eb29-d2c9-4562-a8e4-8e3c82a54c1f admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:12.258773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2eb29-d2c9-4562-a8e4-8e3c82a54c1f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:12.259045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2eb29-d2c9-4562-a8e4-8e3c82a54c1f admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:12.259780 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee2eb29-d2c9-4562-a8e4-8e3c82a54c1f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:12.262833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee2eb29-d2c9-4562-a8e4-8e3c82a54c1f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:12.302109 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1277/2562] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:12.316659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1286/2563] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:17:12 2022] POST /identity/v3/projects => generated 377 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:17:12.336187 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee2eb29-d2c9-4562-a8e4-8e3c82a54c1f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:12.339677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-6ee2eb29-d2c9-4562-a8e4-8e3c82a54c1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:12.364908 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255d2c8a-f552-4123-ba3d-a1f068ed7586 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:12.403190 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255d2c8a-f552-4123-ba3d-a1f068ed7586 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:12.504213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61b77a1-0c09-4c40-b7f3-cb55840e459a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:12.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61b77a1-0c09-4c40-b7f3-cb55840e459a admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:12.504924 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61b77a1-0c09-4c40-b7f3-cb55840e459a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:12.505399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61b77a1-0c09-4c40-b7f3-cb55840e459a admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:12.506108 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61b77a1-0c09-4c40-b7f3-cb55840e459a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:12.508507 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61b77a1-0c09-4c40-b7f3-cb55840e459a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:12.542309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1278/2564] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:17:12 2022] POST /identity/v3/users => generated 295 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:17:12.547462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61b77a1-0c09-4c40-b7f3-cb55840e459a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:12.563398 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee2eb29-d2c9-4562-a8e4-8e3c82a54c1f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:12.572020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cef602b-427a-4b66-b255-0252541a33b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:12.572712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cef602b-427a-4b66-b255-0252541a33b1 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:12.573003 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cef602b-427a-4b66-b255-0252541a33b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:12.573347 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cef602b-427a-4b66-b255-0252541a33b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:12.581299 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61b77a1-0c09-4c40-b7f3-cb55840e459a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:12.589583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cef602b-427a-4b66-b255-0252541a33b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:12.592426 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cef602b-427a-4b66-b255-0252541a33b1 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:12.593810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea85c90-3524-42ab-b19b-527059ce92ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:12.594817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea85c90-3524-42ab-b19b-527059ce92ad admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:12.595278 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea85c90-3524-42ab-b19b-527059ce92ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:12.595784 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea85c90-3524-42ab-b19b-527059ce92ad admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:12.596996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea85c90-3524-42ab-b19b-527059ce92ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:12.600958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea85c90-3524-42ab-b19b-527059ce92ad admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:12.613703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1287/2565] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:17:12.624369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1279/2566] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:12 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:17:12.628909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cef602b-427a-4b66-b255-0252541a33b1 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:12.650784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cef602b-427a-4b66-b255-0252541a33b1 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:12.658483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899e43e6-2921-4032-9fa5-929898feb654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:12.659420 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899e43e6-2921-4032-9fa5-929898feb654 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:12.659905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899e43e6-2921-4032-9fa5-929898feb654 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:12.660386 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899e43e6-2921-4032-9fa5-929898feb654 admin admin] PATH_INFO: `/v3/projects/e19a1a4c25ff4cd6866816e0600b576d/users/6cd5aba99ec8492f9891202300dbaaef/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:12.693011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899e43e6-2921-4032-9fa5-929898feb654 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e19a1a4c25ff4cd6866816e0600b576d, user_id=6cd5aba99ec8492f9891202300dbaaef, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:12.699421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899e43e6-2921-4032-9fa5-929898feb654 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:12.723917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1288/2567] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:17:12 2022] PUT /identity/v3/projects/e19a1a4c25ff4cd6866816e0600b576d/users/6cd5aba99ec8492f9891202300dbaaef/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:12.730034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea85c90-3524-42ab-b19b-527059ce92ad admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:12.753786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea85c90-3524-42ab-b19b-527059ce92ad admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:12.847916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e6f523-f050-418e-912d-9c129436c94a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:12.849470 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e6f523-f050-418e-912d-9c129436c94a admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:12.849680 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e6f523-f050-418e-912d-9c129436c94a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:12.850152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e6f523-f050-418e-912d-9c129436c94a admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:12.850872 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e6f523-f050-418e-912d-9c129436c94a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:12.853713 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e6f523-f050-418e-912d-9c129436c94a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:12.874255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1280/2568] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:12 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:17:12.875733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899e43e6-2921-4032-9fa5-929898feb654 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:12.896268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899e43e6-2921-4032-9fa5-929898feb654 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:12.904058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc960f74-e906-403f-bbb7-340aea4aa24f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:12.904857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc960f74-e906-403f-bbb7-340aea4aa24f admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:12.905287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc960f74-e906-403f-bbb7-340aea4aa24f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:12.905791 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc960f74-e906-403f-bbb7-340aea4aa24f admin admin] PATH_INFO: `/v3/projects/e19a1a4c25ff4cd6866816e0600b576d/users/6cd5aba99ec8492f9891202300dbaaef/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:12.909370 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc960f74-e906-403f-bbb7-340aea4aa24f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e19a1a4c25ff4cd6866816e0600b576d, user_id=6cd5aba99ec8492f9891202300dbaaef, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:12.914187 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc960f74-e906-403f-bbb7-340aea4aa24f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:12.931289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1289/2569] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:17:12 2022] PUT /identity/v3/projects/e19a1a4c25ff4cd6866816e0600b576d/users/6cd5aba99ec8492f9891202300dbaaef/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:13.184489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e6f523-f050-418e-912d-9c129436c94a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:13.186729 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-84e6f523-f050-418e-912d-9c129436c94a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:13.209435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e6f523-f050-418e-912d-9c129436c94a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:13.306362 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:13.351797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:13.352911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:13.352911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:13.353277 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] PATH_INFO: `/v3/projects/35a46b7fdfe24f2d85cf7a738da77c18` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:13.355458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] RBAC: Authorizing `identity:delete_project(project_id=35a46b7fdfe24f2d85cf7a738da77c18)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:13.358320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:13.452181 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35a46b7fdfe24f2d85cf7a738da77c18', 'request_id': 'req-df1e4d75-c261-4aec-bafa-61e4c0bf826c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:13.460876 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35a46b7fdfe24f2d85cf7a738da77c18', 'request_id': 'req-df1e4d75-c261-4aec-bafa-61e4c0bf826c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:13.480683 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35a46b7fdfe24f2d85cf7a738da77c18', 'request_id': 'req-df1e4d75-c261-4aec-bafa-61e4c0bf826c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:13.535983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] The token cache is being invalidate because project 35a46b7fdfe24f2d85cf7a738da77c18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:17:13.539479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:13.544363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:13.557590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1281/2570] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:17:13 2022] DELETE /identity/v3/projects/35a46b7fdfe24f2d85cf7a738da77c18 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:13.562939 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc960f74-e906-403f-bbb7-340aea4aa24f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:13.566782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-cc960f74-e906-403f-bbb7-340aea4aa24f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:13.588785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc960f74-e906-403f-bbb7-340aea4aa24f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:13.725933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:13.766750 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6073776-8742-4716-acd5-2e97c4f6a43f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:13.767848 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6073776-8742-4716-acd5-2e97c4f6a43f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:13.768299 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6073776-8742-4716-acd5-2e97c4f6a43f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:13.768740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6073776-8742-4716-acd5-2e97c4f6a43f admin admin] PATH_INFO: `/v3/users/098d180d26e54458935c84063e4c416d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:13.771206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6073776-8742-4716-acd5-2e97c4f6a43f admin admin] RBAC: Authorizing `identity:delete_user(user_id=098d180d26e54458935c84063e4c416d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:13.776100 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6073776-8742-4716-acd5-2e97c4f6a43f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:13.962704 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d6073776-8742-4716-acd5-2e97c4f6a43f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '098d180d26e54458935c84063e4c416d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:14.000169 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d6073776-8742-4716-acd5-2e97c4f6a43f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '098d180d26e54458935c84063e4c416d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:14.003446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d6073776-8742-4716-acd5-2e97c4f6a43f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '098d180d26e54458935c84063e4c416d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:14.050522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d6073776-8742-4716-acd5-2e97c4f6a43f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '098d180d26e54458935c84063e4c416d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:14.059861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d6073776-8742-4716-acd5-2e97c4f6a43f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '098d180d26e54458935c84063e4c416d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:14.115866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1290/2571] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:17:13 2022] DELETE /identity/v3/users/098d180d26e54458935c84063e4c416d => generated 0 bytes in 548 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:14.704456 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:14.739855 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1e4d75-c261-4aec-bafa-61e4c0bf826c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:14.840146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6617d737-0e36-4460-b068-af945cf23be4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:14.840976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6617d737-0e36-4460-b068-af945cf23be4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:14.842087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6617d737-0e36-4460-b068-af945cf23be4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:14.842087 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6617d737-0e36-4460-b068-af945cf23be4 admin admin] PATH_INFO: `/v3/projects/c1ce1171c40e4d9099b2ab4e334b9738` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:14.843266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6617d737-0e36-4460-b068-af945cf23be4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1ce1171c40e4d9099b2ab4e334b9738)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:14.845414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6617d737-0e36-4460-b068-af945cf23be4 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:14.922052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6617d737-0e36-4460-b068-af945cf23be4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1ce1171c40e4d9099b2ab4e334b9738', 'request_id': 'req-6617d737-0e36-4460-b068-af945cf23be4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:14.933217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6617d737-0e36-4460-b068-af945cf23be4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1ce1171c40e4d9099b2ab4e334b9738', 'request_id': 'req-6617d737-0e36-4460-b068-af945cf23be4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:14.946098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6617d737-0e36-4460-b068-af945cf23be4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1ce1171c40e4d9099b2ab4e334b9738', 'request_id': 'req-6617d737-0e36-4460-b068-af945cf23be4'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:14.956091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:14.965236 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6617d737-0e36-4460-b068-af945cf23be4 admin admin] The token cache is being invalidate because project c1ce1171c40e4d9099b2ab4e334b9738 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:17:14.966355 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6617d737-0e36-4460-b068-af945cf23be4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:14.969061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6617d737-0e36-4460-b068-af945cf23be4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:14.972081 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1282/2572] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:17:14 2022] DELETE /identity/v3/projects/c1ce1171c40e4d9099b2ab4e334b9738 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:14.978433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a33292b-b9eb-4a23-b5ec-948278e0f07d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:14.978936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a33292b-b9eb-4a23-b5ec-948278e0f07d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:14.979468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a33292b-b9eb-4a23-b5ec-948278e0f07d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:15.007560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-3a33292b-b9eb-4a23-b5ec-948278e0f07d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:15.009283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-3a33292b-b9eb-4a23-b5ec-948278e0f07d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:15.132034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1291/2573] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:15.141624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a09df6-8257-404e-a277-adaa6879e791 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:15.142587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a09df6-8257-404e-a277-adaa6879e791 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:15.143107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a09df6-8257-404e-a277-adaa6879e791 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:15.182542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-26a09df6-8257-404e-a277-adaa6879e791 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:15.184352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-26a09df6-8257-404e-a277-adaa6879e791 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:15.195734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1283/2574] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:15.200602 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a33292b-b9eb-4a23-b5ec-948278e0f07d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:15.214129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a33292b-b9eb-4a23-b5ec-948278e0f07d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:15.238655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8044013-092d-413e-b9bb-c18f7169314e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:15.239311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8044013-092d-413e-b9bb-c18f7169314e admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:15.239675 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8044013-092d-413e-b9bb-c18f7169314e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:15.239950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8044013-092d-413e-b9bb-c18f7169314e admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:15.240678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8044013-092d-413e-b9bb-c18f7169314e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:17:15.240999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8044013-092d-413e-b9bb-c18f7169314e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:15.242661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8044013-092d-413e-b9bb-c18f7169314e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:15.266495 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1292/2575] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:17:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:17:15.268083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a09df6-8257-404e-a277-adaa6879e791 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:15.284298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a09df6-8257-404e-a277-adaa6879e791 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:15.293791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e187848a-f54f-4b01-ad00-b7afdcb28feb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:15.294650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e187848a-f54f-4b01-ad00-b7afdcb28feb admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:15.295035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e187848a-f54f-4b01-ad00-b7afdcb28feb admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:15.295268 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e187848a-f54f-4b01-ad00-b7afdcb28feb admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:15.296114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e187848a-f54f-4b01-ad00-b7afdcb28feb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:15.299200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e187848a-f54f-4b01-ad00-b7afdcb28feb admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:15.374584 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1284/2576] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:17:15 2022] POST /identity/v3/projects => generated 357 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:17:15.376005 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8044013-092d-413e-b9bb-c18f7169314e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:15.391753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8044013-092d-413e-b9bb-c18f7169314e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:15.405438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7c2687-0f2c-4d2a-bc4d-a3522a625bbc None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:15.405720 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7c2687-0f2c-4d2a-bc4d-a3522a625bbc None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:15.406029 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7c2687-0f2c-4d2a-bc4d-a3522a625bbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:15.445305 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-fc7c2687-0f2c-4d2a-bc4d-a3522a625bbc None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:15.447060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-fc7c2687-0f2c-4d2a-bc4d-a3522a625bbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:15.580159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:15.595695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1285/2577] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:17:15 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:17:15.625515 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7c2687-0f2c-4d2a-bc4d-a3522a625bbc None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:15.628905 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-fc7c2687-0f2c-4d2a-bc4d-a3522a625bbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:15.643487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9a46b5-6fd3-4c1f-909f-5b54edc2d9f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:15.647140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9a46b5-6fd3-4c1f-909f-5b54edc2d9f7 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:15.647525 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9a46b5-6fd3-4c1f-909f-5b54edc2d9f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:15.647805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9a46b5-6fd3-4c1f-909f-5b54edc2d9f7 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:15.650518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9a46b5-6fd3-4c1f-909f-5b54edc2d9f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:15.657297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9a46b5-6fd3-4c1f-909f-5b54edc2d9f7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:15.719229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1293/2578] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:17:15 2022] POST /identity/v3/users => generated 279 bytes in 344 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:17:15.726235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9a46b5-6fd3-4c1f-909f-5b54edc2d9f7 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:15.762659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9a46b5-6fd3-4c1f-909f-5b54edc2d9f7 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:15.777925 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c3e397-8847-4b2b-b4f1-e64fa454a4b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:15.778927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c3e397-8847-4b2b-b4f1-e64fa454a4b5 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:15.779549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c3e397-8847-4b2b-b4f1-e64fa454a4b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:15.779939 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c3e397-8847-4b2b-b4f1-e64fa454a4b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:15.780881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c3e397-8847-4b2b-b4f1-e64fa454a4b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:15.782858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c3e397-8847-4b2b-b4f1-e64fa454a4b5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:15.843437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1294/2579] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:15 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:17:15.849725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c3e397-8847-4b2b-b4f1-e64fa454a4b5 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:15.881165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c3e397-8847-4b2b-b4f1-e64fa454a4b5 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:15.887389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7c2687-0f2c-4d2a-bc4d-a3522a625bbc None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:15.888226 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d54d61f-b927-442a-a56c-893dd0b3ad5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:15.888478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d54d61f-b927-442a-a56c-893dd0b3ad5a admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:15.888949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d54d61f-b927-442a-a56c-893dd0b3ad5a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:15.889321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d54d61f-b927-442a-a56c-893dd0b3ad5a admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:15.890251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d54d61f-b927-442a-a56c-893dd0b3ad5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:15.892402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d54d61f-b927-442a-a56c-893dd0b3ad5a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:15.913716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1295/2580] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:15 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:17:15.917175 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c52076-6cf5-473b-ad2c-616913170612 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:15.918799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c52076-6cf5-473b-ad2c-616913170612 service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:15.919492 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c52076-6cf5-473b-ad2c-616913170612 service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:15.919980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c52076-6cf5-473b-ad2c-616913170612 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:15.921617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d54d61f-b927-442a-a56c-893dd0b3ad5a admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:15.944368 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d54d61f-b927-442a-a56c-893dd0b3ad5a admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:15.961911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f5376f-fc90-433f-8387-0e82d1a22010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:15.963547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f5376f-fc90-433f-8387-0e82d1a22010 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:15.963880 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f5376f-fc90-433f-8387-0e82d1a22010 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:15.964052 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c52076-6cf5-473b-ad2c-616913170612 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:15.964367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f5376f-fc90-433f-8387-0e82d1a22010 admin admin] PATH_INFO: `/v3/projects/36cbe05118ef497ba23391f94708efab/users/dac3c1aa5a9a4436acbee2ae0af829c0/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:15.972078 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c52076-6cf5-473b-ad2c-616913170612 service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:15.998572 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f5376f-fc90-433f-8387-0e82d1a22010 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36cbe05118ef497ba23391f94708efab, user_id=dac3c1aa5a9a4436acbee2ae0af829c0, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:16.008327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f5376f-fc90-433f-8387-0e82d1a22010 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:16.010002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1286/2581] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:17:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 01:17:16.050500 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1296/2582] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:17:15 2022] PUT /identity/v3/projects/36cbe05118ef497ba23391f94708efab/users/dac3c1aa5a9a4436acbee2ae0af829c0/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:16.145791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6127d796-6fc1-479e-8120-a6e3d691287f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:16.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6127d796-6fc1-479e-8120-a6e3d691287f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:16.147782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6127d796-6fc1-479e-8120-a6e3d691287f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:16.300074 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-6127d796-6fc1-479e-8120-a6e3d691287f None None] MFA Rules not processed for user `dac3c1aa5a9a4436acbee2ae0af829c0`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:16.301741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6127d796-6fc1-479e-8120-a6e3d691287f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:16.492053 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:16.592699 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1287/2583] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:16 2022] POST /identity/v3/auth/tokens => generated 3224 bytes in 446 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:16.594027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f5376f-fc90-433f-8387-0e82d1a22010 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:16.874077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f5376f-fc90-433f-8387-0e82d1a22010 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:16.914801 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce24f3a-c02b-4bd9-a0e7-279d16911f16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:16.923323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce24f3a-c02b-4bd9-a0e7-279d16911f16 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:16.923861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce24f3a-c02b-4bd9-a0e7-279d16911f16 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:16.924351 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce24f3a-c02b-4bd9-a0e7-279d16911f16 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:16.936034 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce24f3a-c02b-4bd9-a0e7-279d16911f16 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:16.940473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce24f3a-c02b-4bd9-a0e7-279d16911f16 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:17.041424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1297/2584] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:17:16 2022] POST /identity/v3/projects => generated 357 bytes in 447 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:17:17.058380 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6127d796-6fc1-479e-8120-a6e3d691287f None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:17.164709 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6127d796-6fc1-479e-8120-a6e3d691287f None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:17.392953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf9fe4d-18b2-4e05-b2ce-097cdd7c20a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:17.395103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf9fe4d-18b2-4e05-b2ce-097cdd7c20a8 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:17.401352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf9fe4d-18b2-4e05-b2ce-097cdd7c20a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:17.402315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf9fe4d-18b2-4e05-b2ce-097cdd7c20a8 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:17.403262 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf9fe4d-18b2-4e05-b2ce-097cdd7c20a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:17.411770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf9fe4d-18b2-4e05-b2ce-097cdd7c20a8 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:17.488674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1288/2585] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:17:17 2022] POST /identity/v3/users => generated 285 bytes in 431 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:17:17.494720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce24f3a-c02b-4bd9-a0e7-279d16911f16 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:17.528028 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce24f3a-c02b-4bd9-a0e7-279d16911f16 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:17.556119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0da8df-ec68-4ddc-a2ca-c9d87c24d2a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:17.557350 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0da8df-ec68-4ddc-a2ca-c9d87c24d2a4 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:17.557915 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0da8df-ec68-4ddc-a2ca-c9d87c24d2a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:17.558431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0da8df-ec68-4ddc-a2ca-c9d87c24d2a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:17.559456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0da8df-ec68-4ddc-a2ca-c9d87c24d2a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:17.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0da8df-ec68-4ddc-a2ca-c9d87c24d2a4 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:17.629659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1298/2586] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:17 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:17:17.637847 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf9fe4d-18b2-4e05-b2ce-097cdd7c20a8 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:17.678429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf9fe4d-18b2-4e05-b2ce-097cdd7c20a8 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:17.743669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919b9cde-4874-4d8c-98c0-b95d994a8b39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:17.744849 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919b9cde-4874-4d8c-98c0-b95d994a8b39 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:17.745304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919b9cde-4874-4d8c-98c0-b95d994a8b39 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:17.745833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919b9cde-4874-4d8c-98c0-b95d994a8b39 admin admin] PATH_INFO: `/v3/projects/a848aa1e85d143a782aef0fb73c297eb/users/e4258aefd8f841b58ded08c6104dfd13/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:17.848691 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919b9cde-4874-4d8c-98c0-b95d994a8b39 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a848aa1e85d143a782aef0fb73c297eb, user_id=e4258aefd8f841b58ded08c6104dfd13, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:17.862872 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919b9cde-4874-4d8c-98c0-b95d994a8b39 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:17.951736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1289/2587] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:17:17 2022] PUT /identity/v3/projects/a848aa1e85d143a782aef0fb73c297eb/users/e4258aefd8f841b58ded08c6104dfd13/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:17.957908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0da8df-ec68-4ddc-a2ca-c9d87c24d2a4 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:18.000074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0da8df-ec68-4ddc-a2ca-c9d87c24d2a4 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:18.291720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67fc88a-b2f1-464e-8e05-a24b2dfe6bc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:18.293029 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67fc88a-b2f1-464e-8e05-a24b2dfe6bc1 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:18.293637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67fc88a-b2f1-464e-8e05-a24b2dfe6bc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:18.297352 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67fc88a-b2f1-464e-8e05-a24b2dfe6bc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:18.298797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67fc88a-b2f1-464e-8e05-a24b2dfe6bc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:18.302997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67fc88a-b2f1-464e-8e05-a24b2dfe6bc1 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:18.376593 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1299/2588] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:17 2022] GET /identity/v3/roles => generated 1420 bytes in 409 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:17:18.378960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919b9cde-4874-4d8c-98c0-b95d994a8b39 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:18.412230 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919b9cde-4874-4d8c-98c0-b95d994a8b39 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:18.448764 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500ed58e-f259-4cde-ac7d-6ba9b44a3e77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:18.452077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500ed58e-f259-4cde-ac7d-6ba9b44a3e77 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:18.454941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500ed58e-f259-4cde-ac7d-6ba9b44a3e77 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:18.456572 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500ed58e-f259-4cde-ac7d-6ba9b44a3e77 admin admin] PATH_INFO: `/v3/projects/a848aa1e85d143a782aef0fb73c297eb/users/e4258aefd8f841b58ded08c6104dfd13/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:18.470439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500ed58e-f259-4cde-ac7d-6ba9b44a3e77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a848aa1e85d143a782aef0fb73c297eb, user_id=e4258aefd8f841b58ded08c6104dfd13, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:18.482856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500ed58e-f259-4cde-ac7d-6ba9b44a3e77 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:18.508653 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1290/2589] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:17:18 2022] PUT /identity/v3/projects/a848aa1e85d143a782aef0fb73c297eb/users/e4258aefd8f841b58ded08c6104dfd13/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:18.528400 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5ed3e4-1576-43d8-8c7a-2e5f10bd3f9a None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:18.529109 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5ed3e4-1576-43d8-8c7a-2e5f10bd3f9a None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:18.529419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5ed3e4-1576-43d8-8c7a-2e5f10bd3f9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:18.552677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:18.650935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-2c5ed3e4-1576-43d8-8c7a-2e5f10bd3f9a None None] MFA Rules not processed for user `e4258aefd8f841b58ded08c6104dfd13`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:18.655927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2c5ed3e4-1576-43d8-8c7a-2e5f10bd3f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:18.906773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1300/2590] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:18 2022] POST /identity/v3/auth/tokens => generated 3291 bytes in 376 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:18.940659 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500ed58e-f259-4cde-ac7d-6ba9b44a3e77 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:18.942489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-500ed58e-f259-4cde-ac7d-6ba9b44a3e77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:19.113944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500ed58e-f259-4cde-ac7d-6ba9b44a3e77 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:19.135395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5d240-414e-4d89-9c71-7dc25fd899be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:19.139534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5d240-414e-4d89-9c71-7dc25fd899be service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:19.140490 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5d240-414e-4d89-9c71-7dc25fd899be service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:19.141291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5d240-414e-4d89-9c71-7dc25fd899be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:19.171189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f5d240-414e-4d89-9c71-7dc25fd899be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:19.175476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f5d240-414e-4d89-9c71-7dc25fd899be service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:19.208931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1291/2591] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:18 2022] GET /identity/v3/auth/tokens => generated 3291 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:17:19.404265 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5ed3e4-1576-43d8-8c7a-2e5f10bd3f9a None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:19.444806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5ed3e4-1576-43d8-8c7a-2e5f10bd3f9a None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:19.451488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313aece3-4a57-4bcf-9b5e-860c6fa1304a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:19.452646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313aece3-4a57-4bcf-9b5e-860c6fa1304a service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:19.453248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313aece3-4a57-4bcf-9b5e-860c6fa1304a service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:19.453788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313aece3-4a57-4bcf-9b5e-860c6fa1304a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:19.660798 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313aece3-4a57-4bcf-9b5e-860c6fa1304a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:19.665378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313aece3-4a57-4bcf-9b5e-860c6fa1304a service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:19.692672 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1301/2592] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:19 2022] GET /identity/v3/auth/tokens => generated 3224 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:17:23.928756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d608315e-2de1-4fec-9885-d9526fd576b2 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:23.930276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d608315e-2de1-4fec-9885-d9526fd576b2 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:23.931003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d608315e-2de1-4fec-9885-d9526fd576b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:24.053395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-d608315e-2de1-4fec-9885-d9526fd576b2 None None] MFA Rules not processed for user `6cd5aba99ec8492f9891202300dbaaef`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:24.058063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d608315e-2de1-4fec-9885-d9526fd576b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:24.214430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:24.352230 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1292/2593] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:23 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 426 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:24.397381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313aece3-4a57-4bcf-9b5e-860c6fa1304a service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:24.400209 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-313aece3-4a57-4bcf-9b5e-860c6fa1304a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:24.569955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313aece3-4a57-4bcf-9b5e-860c6fa1304a service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:24.597075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d78ae5-a4dc-40fa-a5a3-f9bc40f1bc75 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:24.598203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d78ae5-a4dc-40fa-a5a3-f9bc40f1bc75 service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:24.599040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d78ae5-a4dc-40fa-a5a3-f9bc40f1bc75 service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:24.599606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d78ae5-a4dc-40fa-a5a3-f9bc40f1bc75 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:24.609483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-35d78ae5-a4dc-40fa-a5a3-f9bc40f1bc75 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:24.645688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:24.713285 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d78ae5-a4dc-40fa-a5a3-f9bc40f1bc75 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:24.715080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d78ae5-a4dc-40fa-a5a3-f9bc40f1bc75 service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:24.729278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1302/2594] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:24 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:17:34.333013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=13 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:34.722871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=17 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:39.289960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d608315e-2de1-4fec-9885-d9526fd576b2 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:39.366339 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d608315e-2de1-4fec-9885-d9526fd576b2 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:39.374599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:39.375197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:39.375582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:39.375885 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] PATH_INFO: `/v3/users/dac3c1aa5a9a4436acbee2ae0af829c0` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:39.377237 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=dac3c1aa5a9a4436acbee2ae0af829c0)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:39.379199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:39.448157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dac3c1aa5a9a4436acbee2ae0af829c0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:39.448860 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dac3c1aa5a9a4436acbee2ae0af829c0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:39.457172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dac3c1aa5a9a4436acbee2ae0af829c0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:39.476325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dac3c1aa5a9a4436acbee2ae0af829c0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:39.485606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dac3c1aa5a9a4436acbee2ae0af829c0'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:39.493357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1293/2595] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:17:39 2022] DELETE /identity/v3/users/dac3c1aa5a9a4436acbee2ae0af829c0 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:39.499522 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c7d3a-3ed9-49f0-a64b-914ca5c0773e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:39.500182 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c7d3a-3ed9-49f0-a64b-914ca5c0773e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:39.500309 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c7d3a-3ed9-49f0-a64b-914ca5c0773e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:39.526723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a02c7d3a-3ed9-49f0-a64b-914ca5c0773e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:39.528762 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a02c7d3a-3ed9-49f0-a64b-914ca5c0773e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:39.600237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1303/2596] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:39.611456 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:39.612960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:39.690340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87514f79-3735-428d-aec0-2798bf0a6b3a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:39.694930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b37ac-4b1c-47d4-907b-6025c36b1957 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:39.695795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b37ac-4b1c-47d4-907b-6025c36b1957 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:39.696068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b37ac-4b1c-47d4-907b-6025c36b1957 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:39.696336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b37ac-4b1c-47d4-907b-6025c36b1957 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:39.710022 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b37ac-4b1c-47d4-907b-6025c36b1957 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:39.711898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b37ac-4b1c-47d4-907b-6025c36b1957 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:39.725587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1294/2597] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:17:40.051521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c7d3a-3ed9-49f0-a64b-914ca5c0773e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:40.053251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a02c7d3a-3ed9-49f0-a64b-914ca5c0773e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:40.067096 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c7d3a-3ed9-49f0-a64b-914ca5c0773e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:40.073118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:40.075807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:40.076163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:40.076501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] PATH_INFO: `/v3/projects/36cbe05118ef497ba23391f94708efab` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:40.078246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] RBAC: Authorizing `identity:delete_project(project_id=36cbe05118ef497ba23391f94708efab)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:40.080126 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:40.149961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '36cbe05118ef497ba23391f94708efab', 'request_id': 'req-ec827f07-9305-4233-8b6b-50477169ad81'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:40.157452 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '36cbe05118ef497ba23391f94708efab', 'request_id': 'req-ec827f07-9305-4233-8b6b-50477169ad81'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:40.165858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '36cbe05118ef497ba23391f94708efab', 'request_id': 'req-ec827f07-9305-4233-8b6b-50477169ad81'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:40.183216 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] The token cache is being invalidate because project 36cbe05118ef497ba23391f94708efab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:17:40.183563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:40.184475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:40.186945 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1304/2598] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:17:40 2022] DELETE /identity/v3/projects/36cbe05118ef497ba23391f94708efab => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:40.192705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b37ac-4b1c-47d4-907b-6025c36b1957 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:40.203394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b37ac-4b1c-47d4-907b-6025c36b1957 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:40.225965 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:40.276680 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:40.277424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:40.277735 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:40.278071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] PATH_INFO: `/v3/users/e4258aefd8f841b58ded08c6104dfd13` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:40.279606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4258aefd8f841b58ded08c6104dfd13)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:40.287405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:40.341137 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:40.343403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:40.383913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4258aefd8f841b58ded08c6104dfd13'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:40.385476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4258aefd8f841b58ded08c6104dfd13'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:40.400482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4258aefd8f841b58ded08c6104dfd13'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:40.425930 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec827f07-9305-4233-8b6b-50477169ad81 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:40.427180 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4258aefd8f841b58ded08c6104dfd13'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:40.431866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3656d45-7931-48eb-a1ea-690facde6f5d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4160039561434e51a87c6c7a019f2dff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:40.432992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3656d45-7931-48eb-a1ea-690facde6f5d service glance] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:40.433424 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3656d45-7931-48eb-a1ea-690facde6f5d service glance] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:40.433808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3656d45-7931-48eb-a1ea-690facde6f5d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:40.435750 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c3656d45-7931-48eb-a1ea-690facde6f5d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:40.441721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4258aefd8f841b58ded08c6104dfd13'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:40.452341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1295/2599] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:17:40 2022] DELETE /identity/v3/users/e4258aefd8f841b58ded08c6104dfd13 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:40.508119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:40.567379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3656d45-7931-48eb-a1ea-690facde6f5d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:40.569080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3656d45-7931-48eb-a1ea-690facde6f5d service glance] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:40.585737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1305/2600] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:40 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:17:40.668598 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:40.670428 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:40.751051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15f314c-d482-42b0-af36-e5b65f9ad634 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:40.755878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b94363-0901-4ca1-b77d-d5234dbd42c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:40.756525 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b94363-0901-4ca1-b77d-d5234dbd42c3 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:40.756806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b94363-0901-4ca1-b77d-d5234dbd42c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:40.757081 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b94363-0901-4ca1-b77d-d5234dbd42c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:40.759046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d0b94363-0901-4ca1-b77d-d5234dbd42c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:40.769827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b94363-0901-4ca1-b77d-d5234dbd42c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:40.771329 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b94363-0901-4ca1-b77d-d5234dbd42c3 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:40.784289 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1296/2601] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:40 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:17:40.829634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3656d45-7931-48eb-a1ea-690facde6f5d service glance] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:40.832362 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c3656d45-7931-48eb-a1ea-690facde6f5d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:40.859783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3656d45-7931-48eb-a1ea-690facde6f5d service glance] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:40.882678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b94363-0901-4ca1-b77d-d5234dbd42c3 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:40.884606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d0b94363-0901-4ca1-b77d-d5234dbd42c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:40.898336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b94363-0901-4ca1-b77d-d5234dbd42c3 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:40.904558 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a718e52-c9e2-4227-b69b-c961ae126a3f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:40.906085 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a718e52-c9e2-4227-b69b-c961ae126a3f service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:40.906489 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a718e52-c9e2-4227-b69b-c961ae126a3f service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:40.906889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a718e52-c9e2-4227-b69b-c961ae126a3f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:40.909564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9a718e52-c9e2-4227-b69b-c961ae126a3f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:40.945686 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcacba1-d020-4085-88ed-e773e85e7378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:40.946548 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcacba1-d020-4085-88ed-e773e85e7378 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:40.946890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcacba1-d020-4085-88ed-e773e85e7378 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:40.947239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcacba1-d020-4085-88ed-e773e85e7378 admin admin] PATH_INFO: `/v3/projects/a848aa1e85d143a782aef0fb73c297eb` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:40.949397 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcacba1-d020-4085-88ed-e773e85e7378 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a848aa1e85d143a782aef0fb73c297eb)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:40.951480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcacba1-d020-4085-88ed-e773e85e7378 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:40.966877 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:41.004752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a718e52-c9e2-4227-b69b-c961ae126a3f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:41.006227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a718e52-c9e2-4227-b69b-c961ae126a3f service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:41.014524 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-adcacba1-d020-4085-88ed-e773e85e7378 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a848aa1e85d143a782aef0fb73c297eb', 'request_id': 'req-adcacba1-d020-4085-88ed-e773e85e7378'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:41.019066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1297/2602] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:40 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:17:41.023540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-adcacba1-d020-4085-88ed-e773e85e7378 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a848aa1e85d143a782aef0fb73c297eb', 'request_id': 'req-adcacba1-d020-4085-88ed-e773e85e7378'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:41.034624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-adcacba1-d020-4085-88ed-e773e85e7378 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a848aa1e85d143a782aef0fb73c297eb', 'request_id': 'req-adcacba1-d020-4085-88ed-e773e85e7378'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:41.054018 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-adcacba1-d020-4085-88ed-e773e85e7378 admin admin] The token cache is being invalidate because project a848aa1e85d143a782aef0fb73c297eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:17:41.054623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-adcacba1-d020-4085-88ed-e773e85e7378 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:41.055902 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-adcacba1-d020-4085-88ed-e773e85e7378 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:17:41.057961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1306/2603] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:17:40 2022] DELETE /identity/v3/projects/a848aa1e85d143a782aef0fb73c297eb => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:41.065542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335100d3-398c-4225-bbb4-92dd00caf9ea None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:41.066016 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335100d3-398c-4225-bbb4-92dd00caf9ea None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:41.066484 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335100d3-398c-4225-bbb4-92dd00caf9ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:41.092776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-335100d3-398c-4225-bbb4-92dd00caf9ea None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:41.094644 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-335100d3-398c-4225-bbb4-92dd00caf9ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:41.199675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1298/2604] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:41.209439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf8900f-4963-4b10-830e-ba87897620cc None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:41.209439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf8900f-4963-4b10-830e-ba87897620cc None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:41.209439 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf8900f-4963-4b10-830e-ba87897620cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:41.238831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-ecf8900f-4963-4b10-830e-ba87897620cc None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:41.240963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ecf8900f-4963-4b10-830e-ba87897620cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:41.254644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1307/2605] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:41.260796 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335100d3-398c-4225-bbb4-92dd00caf9ea None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:41.273930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335100d3-398c-4225-bbb4-92dd00caf9ea None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:41.281271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0eb211-675b-4f66-b86e-12ce8f2c70a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:41.282799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0eb211-675b-4f66-b86e-12ce8f2c70a1 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:41.283328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0eb211-675b-4f66-b86e-12ce8f2c70a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:41.283866 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0eb211-675b-4f66-b86e-12ce8f2c70a1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:41.285028 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0eb211-675b-4f66-b86e-12ce8f2c70a1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:17:41.285516 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0eb211-675b-4f66-b86e-12ce8f2c70a1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:41.288538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0eb211-675b-4f66-b86e-12ce8f2c70a1 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:41.312284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1299/2606] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:17:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:17:41.313360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf8900f-4963-4b10-830e-ba87897620cc None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:41.324344 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf8900f-4963-4b10-830e-ba87897620cc None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:41.329721 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d873fd-31a6-4b97-aa7b-7e57977f3dae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:41.330464 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d873fd-31a6-4b97-aa7b-7e57977f3dae admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:41.330768 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d873fd-31a6-4b97-aa7b-7e57977f3dae admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:41.331089 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d873fd-31a6-4b97-aa7b-7e57977f3dae admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:41.331879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d873fd-31a6-4b97-aa7b-7e57977f3dae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:41.334043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d873fd-31a6-4b97-aa7b-7e57977f3dae admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:41.375547 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1308/2607] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:17:41 2022] POST /identity/v3/projects => generated 383 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:17:41.381719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0eb211-675b-4f66-b86e-12ce8f2c70a1 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:41.395722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0eb211-675b-4f66-b86e-12ce8f2c70a1 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:41.474643 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed598fb-e303-40d3-830d-5ccefb4edf67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:41.475192 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed598fb-e303-40d3-830d-5ccefb4edf67 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:41.475531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed598fb-e303-40d3-830d-5ccefb4edf67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:41.475813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed598fb-e303-40d3-830d-5ccefb4edf67 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:41.476555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed598fb-e303-40d3-830d-5ccefb4edf67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:41.478544 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed598fb-e303-40d3-830d-5ccefb4edf67 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:41.517008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1300/2608] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:17:41 2022] POST /identity/v3/users => generated 292 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:17:41.518560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d873fd-31a6-4b97-aa7b-7e57977f3dae admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:41.534924 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d873fd-31a6-4b97-aa7b-7e57977f3dae admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:41.541857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdef416c-53ca-45f2-b6f2-d671a6fd5255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:41.542894 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdef416c-53ca-45f2-b6f2-d671a6fd5255 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:41.543446 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdef416c-53ca-45f2-b6f2-d671a6fd5255 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:41.543906 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdef416c-53ca-45f2-b6f2-d671a6fd5255 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:41.546436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdef416c-53ca-45f2-b6f2-d671a6fd5255 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:41.558986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdef416c-53ca-45f2-b6f2-d671a6fd5255 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:41.581596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1309/2609] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:41 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:17:41.586378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed598fb-e303-40d3-830d-5ccefb4edf67 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:41.606166 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed598fb-e303-40d3-830d-5ccefb4edf67 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:41.617826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21610b18-e8d9-47e5-8549-cdf6d1272566 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:41.618850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21610b18-e8d9-47e5-8549-cdf6d1272566 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:41.620077 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21610b18-e8d9-47e5-8549-cdf6d1272566 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:41.620741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21610b18-e8d9-47e5-8549-cdf6d1272566 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:41.622245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21610b18-e8d9-47e5-8549-cdf6d1272566 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:41.625344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21610b18-e8d9-47e5-8549-cdf6d1272566 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:41.641122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1301/2610] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:17:41 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:17:41.646523 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdef416c-53ca-45f2-b6f2-d671a6fd5255 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:41.665197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdef416c-53ca-45f2-b6f2-d671a6fd5255 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:41.677191 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a203941-ce92-490e-9d65-8f1ad196f93f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:41.678166 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a203941-ce92-490e-9d65-8f1ad196f93f admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:41.678469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a203941-ce92-490e-9d65-8f1ad196f93f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:41.678722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a203941-ce92-490e-9d65-8f1ad196f93f admin admin] PATH_INFO: `/v3/projects/0f2417d70b00404f8e37d52f694f9a41/users/943af9a17a554646808ac2f2598a9adf/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:41.699001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:41.701132 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a203941-ce92-490e-9d65-8f1ad196f93f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f2417d70b00404f8e37d52f694f9a41, user_id=943af9a17a554646808ac2f2598a9adf, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:41.705393 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a203941-ce92-490e-9d65-8f1ad196f93f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:41.734436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1310/2611] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:17:41 2022] PUT /identity/v3/projects/0f2417d70b00404f8e37d52f694f9a41/users/943af9a17a554646808ac2f2598a9adf/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:17:41.743081 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e3c452-6919-4504-9524-12b0aa2542fe None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:41.743626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e3c452-6919-4504-9524-12b0aa2542fe None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:41.744371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e3c452-6919-4504-9524-12b0aa2542fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:41.782385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-70e3c452-6919-4504-9524-12b0aa2542fe None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:41.784848 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-70e3c452-6919-4504-9524-12b0aa2542fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:41.849833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:41.886623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1302/2612] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:41.899541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a203941-ce92-490e-9d65-8f1ad196f93f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:41.901662 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5a203941-ce92-490e-9d65-8f1ad196f93f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:41.981406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a203941-ce92-490e-9d65-8f1ad196f93f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:41.985298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7663be-801d-4edd-a884-0f479033cc98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:41.985949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7663be-801d-4edd-a884-0f479033cc98 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:41.986224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7663be-801d-4edd-a884-0f479033cc98 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:41.986538 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7663be-801d-4edd-a884-0f479033cc98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:42.009862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7663be-801d-4edd-a884-0f479033cc98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:42.012868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7663be-801d-4edd-a884-0f479033cc98 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:42.032963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1311/2613] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:17:43.653919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f55466-43b2-4877-ba70-bce0a3548651 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:43.654839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f55466-43b2-4877-ba70-bce0a3548651 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:43.654839 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f55466-43b2-4877-ba70-bce0a3548651 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:43.699687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-16f55466-43b2-4877-ba70-bce0a3548651 None None] MFA Rules not processed for user `943af9a17a554646808ac2f2598a9adf`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:43.701628 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-16f55466-43b2-4877-ba70-bce0a3548651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:43.845707 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1303/2614] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:17:43.864266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7663be-801d-4edd-a884-0f479033cc98 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:43.866521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9b7663be-801d-4edd-a884-0f479033cc98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:43.973057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7663be-801d-4edd-a884-0f479033cc98 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:43.977150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25770851-e524-452d-ab55-23b5a244b96f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:43.977835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25770851-e524-452d-ab55-23b5a244b96f service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:43.978148 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25770851-e524-452d-ab55-23b5a244b96f service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:43.978468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25770851-e524-452d-ab55-23b5a244b96f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:43.989303 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25770851-e524-452d-ab55-23b5a244b96f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:43.991124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25770851-e524-452d-ab55-23b5a244b96f service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:44.002058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1312/2615] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:17:43 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:17:44.128536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796b67d7-58a6-49d8-9bb2-d67983f38baa None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:44.128970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796b67d7-58a6-49d8-9bb2-d67983f38baa None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:44.129391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796b67d7-58a6-49d8-9bb2-d67983f38baa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:44.156802 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-796b67d7-58a6-49d8-9bb2-d67983f38baa None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:44.158678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-796b67d7-58a6-49d8-9bb2-d67983f38baa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:44.272215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1304/2616] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Jul 31 01:17:44 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:44.294620 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25770851-e524-452d-ab55-23b5a244b96f service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:17:44.297903 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-25770851-e524-452d-ab55-23b5a244b96f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:44.498934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25770851-e524-452d-ab55-23b5a244b96f service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:17:44.511514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77b2baa-2ad6-4e9b-a4b6-d9cc59039a11 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:17:44.512742 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77b2baa-2ad6-4e9b-a4b6-d9cc59039a11 service swift] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:44.513295 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77b2baa-2ad6-4e9b-a4b6-d9cc59039a11 service swift] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:44.514688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77b2baa-2ad6-4e9b-a4b6-d9cc59039a11 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:44.539204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77b2baa-2ad6-4e9b-a4b6-d9cc59039a11 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:17:44.541021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77b2baa-2ad6-4e9b-a4b6-d9cc59039a11 service swift] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:17:44.573671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1313/2617] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:17:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 01:17:46.801042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9249376e-35e6-455a-83df-935c0223e532 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:46.803524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9249376e-35e6-455a-83df-935c0223e532 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:46.803524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9249376e-35e6-455a-83df-935c0223e532 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:46.840111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-9249376e-35e6-455a-83df-935c0223e532 None None] MFA Rules not processed for user `943af9a17a554646808ac2f2598a9adf`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:46.842380 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9249376e-35e6-455a-83df-935c0223e532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:46.855692 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1305/2618] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:46 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:51.219673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc17b9e9-7308-42d8-a661-84f42e8f6c35 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:17:51.219673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc17b9e9-7308-42d8-a661-84f42e8f6c35 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:17:51.219673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc17b9e9-7308-42d8-a661-84f42e8f6c35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:17:51.262049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-fc17b9e9-7308-42d8-a661-84f42e8f6c35 None None] MFA Rules not processed for user `6a00c618dd284f159d88ad41c0390c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:17:51.263881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fc17b9e9-7308-42d8-a661-84f42e8f6c35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:17:51.302726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:17:51.384264 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1314/2619] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:17:51 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:17:56.830107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=71 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:01.367723 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=13 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:04.474200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9249376e-35e6-455a-83df-935c0223e532 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:04.487897 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9249376e-35e6-455a-83df-935c0223e532 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:04.495156 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4876ec5d-b7ba-4043-8e12-8a4b652d8af6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:04.496398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4876ec5d-b7ba-4043-8e12-8a4b652d8af6 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:04.497100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4876ec5d-b7ba-4043-8e12-8a4b652d8af6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:04.497532 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4876ec5d-b7ba-4043-8e12-8a4b652d8af6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:04.500878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4876ec5d-b7ba-4043-8e12-8a4b652d8af6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:04.656197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4876ec5d-b7ba-4043-8e12-8a4b652d8af6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:04.659704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4876ec5d-b7ba-4043-8e12-8a4b652d8af6 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:04.680916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1306/2620] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:04 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:18:05.921278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee7d8ab-b335-4eca-b28b-5ca1104c8cff None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:05.922132 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee7d8ab-b335-4eca-b28b-5ca1104c8cff None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:05.922132 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee7d8ab-b335-4eca-b28b-5ca1104c8cff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:05.947538 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-9ee7d8ab-b335-4eca-b28b-5ca1104c8cff None None] MFA Rules not processed for user `8cdffe91785e42d798b931082969598e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:05.949578 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9ee7d8ab-b335-4eca-b28b-5ca1104c8cff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:05.959431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1315/2621] 10.222.0.42 () {58 vars in 967 bytes} [Sun Jul 31 01:18:05 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:18:05.993369 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4876ec5d-b7ba-4043-8e12-8a4b652d8af6 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:06.010691 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4876ec5d-b7ba-4043-8e12-8a4b652d8af6 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:06.017411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8501e582-f065-4ba5-a71f-68f7eba99268 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c11f1dd5a6b042c9b864a5861d0f3569', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:06.019143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8501e582-f065-4ba5-a71f-68f7eba99268 service swift] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:06.019674 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8501e582-f065-4ba5-a71f-68f7eba99268 service swift] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:06.020121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8501e582-f065-4ba5-a71f-68f7eba99268 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:06.033751 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8501e582-f065-4ba5-a71f-68f7eba99268 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:06.036187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8501e582-f065-4ba5-a71f-68f7eba99268 service swift] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:06.059627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1307/2622] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Jul 31 01:18:05 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 31 01:18:07.353831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee7d8ab-b335-4eca-b28b-5ca1104c8cff None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:07.355626 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9ee7d8ab-b335-4eca-b28b-5ca1104c8cff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:07.368434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee7d8ab-b335-4eca-b28b-5ca1104c8cff None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:07.372318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6994d99-325f-4b25-9837-c0f242589177 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:07.378332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6994d99-325f-4b25-9837-c0f242589177 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:07.378332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6994d99-325f-4b25-9837-c0f242589177 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:07.378332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6994d99-325f-4b25-9837-c0f242589177 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:07.378332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-c6994d99-325f-4b25-9837-c0f242589177 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:07.393927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6994d99-325f-4b25-9837-c0f242589177 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:07.395702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6994d99-325f-4b25-9837-c0f242589177 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:07.413519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1316/2623] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:07 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:18:16.046357 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=40 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:17.403293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=9 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:25.862851 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8501e582-f065-4ba5-a71f-68f7eba99268 service swift] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:25.875958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8501e582-f065-4ba5-a71f-68f7eba99268 service swift] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:25.880745 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f32ba1-c16c-4de0-9d5c-029dc8407d56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:25.881543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f32ba1-c16c-4de0-9d5c-029dc8407d56 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:25.881938 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f32ba1-c16c-4de0-9d5c-029dc8407d56 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:25.882255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f32ba1-c16c-4de0-9d5c-029dc8407d56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:25.884250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-81f32ba1-c16c-4de0-9d5c-029dc8407d56 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:25.974056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f32ba1-c16c-4de0-9d5c-029dc8407d56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:25.975614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f32ba1-c16c-4de0-9d5c-029dc8407d56 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:25.987973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1308/2624] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:25 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:18:27.609940 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6994d99-325f-4b25-9837-c0f242589177 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:27.634997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6994d99-325f-4b25-9837-c0f242589177 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:27.640599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec0f6c1-ec0b-42f4-bf38-74454da5e61d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:27.641598 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec0f6c1-ec0b-42f4-bf38-74454da5e61d service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:27.642117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec0f6c1-ec0b-42f4-bf38-74454da5e61d service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:27.642643 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec0f6c1-ec0b-42f4-bf38-74454da5e61d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:27.645272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-cec0f6c1-ec0b-42f4-bf38-74454da5e61d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:27.664305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec0f6c1-ec0b-42f4-bf38-74454da5e61d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:27.666499 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec0f6c1-ec0b-42f4-bf38-74454da5e61d service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:27.685914 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1317/2625] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:18:32.590693 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f32ba1-c16c-4de0-9d5c-029dc8407d56 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:32.593141 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-81f32ba1-c16c-4de0-9d5c-029dc8407d56 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:32.613410 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f32ba1-c16c-4de0-9d5c-029dc8407d56 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:32.619155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638aa211-885b-4afc-b8f9-227a533860cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:32.621113 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638aa211-885b-4afc-b8f9-227a533860cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:32.621511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638aa211-885b-4afc-b8f9-227a533860cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:32.621899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638aa211-885b-4afc-b8f9-227a533860cf admin admin] PATH_INFO: `/v3/users/20df8d74c87143dd81dbaae76b3c2b62` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:32.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638aa211-885b-4afc-b8f9-227a533860cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=20df8d74c87143dd81dbaae76b3c2b62)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:32.642122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638aa211-885b-4afc-b8f9-227a533860cf admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:32.706820 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-638aa211-885b-4afc-b8f9-227a533860cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20df8d74c87143dd81dbaae76b3c2b62'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:32.707738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-638aa211-885b-4afc-b8f9-227a533860cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20df8d74c87143dd81dbaae76b3c2b62'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:32.718756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-638aa211-885b-4afc-b8f9-227a533860cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20df8d74c87143dd81dbaae76b3c2b62'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:32.736966 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-638aa211-885b-4afc-b8f9-227a533860cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20df8d74c87143dd81dbaae76b3c2b62'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:32.746387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-638aa211-885b-4afc-b8f9-227a533860cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20df8d74c87143dd81dbaae76b3c2b62'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:32.754127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1309/2626] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:18:32 2022] DELETE /identity/v3/users/20df8d74c87143dd81dbaae76b3c2b62 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:32.997746 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec0f6c1-ec0b-42f4-bf38-74454da5e61d service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:33.017140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec0f6c1-ec0b-42f4-bf38-74454da5e61d service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:33.101814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da747c3c-a02f-46ee-ab19-05337a42973b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:33.102489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da747c3c-a02f-46ee-ab19-05337a42973b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:33.102790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da747c3c-a02f-46ee-ab19-05337a42973b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:33.103057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da747c3c-a02f-46ee-ab19-05337a42973b admin admin] PATH_INFO: `/v3/projects/b94a935febe14fc5b0bda4ca88f78ca8` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:33.115708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da747c3c-a02f-46ee-ab19-05337a42973b admin admin] RBAC: Authorizing `identity:delete_project(project_id=b94a935febe14fc5b0bda4ca88f78ca8)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:33.117720 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da747c3c-a02f-46ee-ab19-05337a42973b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:33.186728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-da747c3c-a02f-46ee-ab19-05337a42973b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b94a935febe14fc5b0bda4ca88f78ca8', 'request_id': 'req-da747c3c-a02f-46ee-ab19-05337a42973b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:33.207925 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-da747c3c-a02f-46ee-ab19-05337a42973b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b94a935febe14fc5b0bda4ca88f78ca8', 'request_id': 'req-da747c3c-a02f-46ee-ab19-05337a42973b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:33.207925 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-da747c3c-a02f-46ee-ab19-05337a42973b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b94a935febe14fc5b0bda4ca88f78ca8', 'request_id': 'req-da747c3c-a02f-46ee-ab19-05337a42973b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:33.223206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-da747c3c-a02f-46ee-ab19-05337a42973b admin admin] The token cache is being invalidate because project b94a935febe14fc5b0bda4ca88f78ca8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:18:33.223629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-da747c3c-a02f-46ee-ab19-05337a42973b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:33.224584 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-da747c3c-a02f-46ee-ab19-05337a42973b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:33.226417 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1318/2627] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:18:32 2022] DELETE /identity/v3/projects/b94a935febe14fc5b0bda4ca88f78ca8 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:33.232139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f101c0d-a771-4b15-9001-e8c5a4ef2c0e None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:33.232469 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f101c0d-a771-4b15-9001-e8c5a4ef2c0e None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:33.232784 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f101c0d-a771-4b15-9001-e8c5a4ef2c0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:33.255650 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-3f101c0d-a771-4b15-9001-e8c5a4ef2c0e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:33.257393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3f101c0d-a771-4b15-9001-e8c5a4ef2c0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:33.326760 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:33.345485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1310/2628] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:33.356466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32da3b7b-0644-452e-90c9-e71277eca669 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:33.356752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32da3b7b-0644-452e-90c9-e71277eca669 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:33.357030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32da3b7b-0644-452e-90c9-e71277eca669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:33.380116 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-32da3b7b-0644-452e-90c9-e71277eca669 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:33.381659 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-32da3b7b-0644-452e-90c9-e71277eca669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:33.390919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1319/2629] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:33.395802 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f101c0d-a771-4b15-9001-e8c5a4ef2c0e None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:33.406503 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f101c0d-a771-4b15-9001-e8c5a4ef2c0e None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:33.411500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e6fbd2-4853-44b4-a23e-9b7ab8afadff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:33.412146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e6fbd2-4853-44b4-a23e-9b7ab8afadff admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:33.412619 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e6fbd2-4853-44b4-a23e-9b7ab8afadff admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:33.412913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e6fbd2-4853-44b4-a23e-9b7ab8afadff admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:33.413479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e6fbd2-4853-44b4-a23e-9b7ab8afadff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:18:33.413782 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e6fbd2-4853-44b4-a23e-9b7ab8afadff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:33.415480 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e6fbd2-4853-44b4-a23e-9b7ab8afadff admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:33.430171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1311/2630] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:18:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:18:33.435886 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32da3b7b-0644-452e-90c9-e71277eca669 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:33.446435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32da3b7b-0644-452e-90c9-e71277eca669 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:33.451810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0a25fe-4fd0-412e-b7ee-a60182898aa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:33.452456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0a25fe-4fd0-412e-b7ee-a60182898aa9 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:33.452739 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0a25fe-4fd0-412e-b7ee-a60182898aa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:33.453040 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0a25fe-4fd0-412e-b7ee-a60182898aa9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:33.453748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0a25fe-4fd0-412e-b7ee-a60182898aa9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:33.455799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0a25fe-4fd0-412e-b7ee-a60182898aa9 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:33.502414 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1320/2631] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:18:33 2022] POST /identity/v3/projects => generated 403 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:18:33.507076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e6fbd2-4853-44b4-a23e-9b7ab8afadff admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:33.518418 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e6fbd2-4853-44b4-a23e-9b7ab8afadff admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:33.593274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f3030e-5f2b-428c-a38a-efebb74cad50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:33.594145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f3030e-5f2b-428c-a38a-efebb74cad50 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:33.594623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f3030e-5f2b-428c-a38a-efebb74cad50 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:33.594989 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f3030e-5f2b-428c-a38a-efebb74cad50 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:33.595841 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f3030e-5f2b-428c-a38a-efebb74cad50 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:33.597962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f3030e-5f2b-428c-a38a-efebb74cad50 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:33.626708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1312/2632] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:18:33 2022] POST /identity/v3/users => generated 302 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:18:33.631911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0a25fe-4fd0-412e-b7ee-a60182898aa9 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:33.650817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0a25fe-4fd0-412e-b7ee-a60182898aa9 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:33.660490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52720c7-eece-4e2a-8b98-d723d9dee439 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:33.661604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52720c7-eece-4e2a-8b98-d723d9dee439 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:33.662098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52720c7-eece-4e2a-8b98-d723d9dee439 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:33.662584 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52720c7-eece-4e2a-8b98-d723d9dee439 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:33.664290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52720c7-eece-4e2a-8b98-d723d9dee439 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:33.668126 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52720c7-eece-4e2a-8b98-d723d9dee439 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:33.690892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1321/2633] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:33 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:33.695631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f3030e-5f2b-428c-a38a-efebb74cad50 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:33.707168 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f3030e-5f2b-428c-a38a-efebb74cad50 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:33.712527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6865f8c6-3495-4009-87ee-6bedba85a3a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:33.713166 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6865f8c6-3495-4009-87ee-6bedba85a3a4 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:33.713445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6865f8c6-3495-4009-87ee-6bedba85a3a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:33.714237 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6865f8c6-3495-4009-87ee-6bedba85a3a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:33.714963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6865f8c6-3495-4009-87ee-6bedba85a3a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:33.716682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6865f8c6-3495-4009-87ee-6bedba85a3a4 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:33.731114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1313/2634] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:33 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:33.737314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52720c7-eece-4e2a-8b98-d723d9dee439 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:33.755150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52720c7-eece-4e2a-8b98-d723d9dee439 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:33.762655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fbefde-450e-4754-8e6d-899d93dfaed3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:33.763716 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fbefde-450e-4754-8e6d-899d93dfaed3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:33.764105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fbefde-450e-4754-8e6d-899d93dfaed3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:33.764478 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fbefde-450e-4754-8e6d-899d93dfaed3 admin admin] PATH_INFO: `/v3/projects/172e4106afc245e3aea7743850e12de3/users/066c282e72aa453fb05eb23b80c1fa11/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:33.786389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fbefde-450e-4754-8e6d-899d93dfaed3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=172e4106afc245e3aea7743850e12de3, user_id=066c282e72aa453fb05eb23b80c1fa11, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:33.790677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fbefde-450e-4754-8e6d-899d93dfaed3 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:33.795540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:33.813655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1322/2635] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:18:33 2022] PUT /identity/v3/projects/172e4106afc245e3aea7743850e12de3/users/066c282e72aa453fb05eb23b80c1fa11/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:33.819643 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b9a045-b297-4897-b28b-54f4c3c30286 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:33.820056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b9a045-b297-4897-b28b-54f4c3c30286 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:33.820499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b9a045-b297-4897-b28b-54f4c3c30286 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:33.842293 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-32b9a045-b297-4897-b28b-54f4c3c30286 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:33.843738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-32b9a045-b297-4897-b28b-54f4c3c30286 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:33.916352 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1314/2636] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:33.928846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fbefde-450e-4754-8e6d-899d93dfaed3 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:33.931287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-08fbefde-450e-4754-8e6d-899d93dfaed3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:34.011985 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fbefde-450e-4754-8e6d-899d93dfaed3 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:34.019072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ad7f9-3cd3-4256-b36f-a21dc598ff8b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:34.020583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ad7f9-3cd3-4256-b36f-a21dc598ff8b service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:34.021170 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ad7f9-3cd3-4256-b36f-a21dc598ff8b service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:34.022058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ad7f9-3cd3-4256-b36f-a21dc598ff8b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:34.035342 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94ad7f9-3cd3-4256-b36f-a21dc598ff8b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:34.036873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94ad7f9-3cd3-4256-b36f-a21dc598ff8b service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:34.047886 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1323/2637] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:18:34.881037 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0292d6-2fed-4d16-bd1d-66acf41a8ac4 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:34.881662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0292d6-2fed-4d16-bd1d-66acf41a8ac4 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:34.882160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0292d6-2fed-4d16-bd1d-66acf41a8ac4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:34.923015 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-3d0292d6-2fed-4d16-bd1d-66acf41a8ac4 None None] MFA Rules not processed for user `066c282e72aa453fb05eb23b80c1fa11`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:34.924601 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3d0292d6-2fed-4d16-bd1d-66acf41a8ac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:34.996522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:35.029080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1315/2638] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:34 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:35.034896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ad7f9-3cd3-4256-b36f-a21dc598ff8b service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:35.053865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ad7f9-3cd3-4256-b36f-a21dc598ff8b service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:35.063093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deee3c2-9c12-4ee2-9a3a-e8da2d1b4775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:35.064205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deee3c2-9c12-4ee2-9a3a-e8da2d1b4775 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:35.064631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deee3c2-9c12-4ee2-9a3a-e8da2d1b4775 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:35.065062 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deee3c2-9c12-4ee2-9a3a-e8da2d1b4775 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:35.066315 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7deee3c2-9c12-4ee2-9a3a-e8da2d1b4775 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:35.070244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7deee3c2-9c12-4ee2-9a3a-e8da2d1b4775 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:35.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1324/2639] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:18:35 2022] POST /identity/v3/projects => generated 405 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:18:35.118716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0292d6-2fed-4d16-bd1d-66acf41a8ac4 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:35.132560 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0292d6-2fed-4d16-bd1d-66acf41a8ac4 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:35.206983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3ffee6-9298-44be-9bcd-9fdc2527e7ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:35.207656 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3ffee6-9298-44be-9bcd-9fdc2527e7ed admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:35.207954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3ffee6-9298-44be-9bcd-9fdc2527e7ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:35.208269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3ffee6-9298-44be-9bcd-9fdc2527e7ed admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:35.208928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3ffee6-9298-44be-9bcd-9fdc2527e7ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:35.210923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3ffee6-9298-44be-9bcd-9fdc2527e7ed admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:35.237318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1316/2640] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:18:35 2022] POST /identity/v3/users => generated 309 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:18:35.243113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deee3c2-9c12-4ee2-9a3a-e8da2d1b4775 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:35.254436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deee3c2-9c12-4ee2-9a3a-e8da2d1b4775 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:35.259898 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145acb39-df28-46e1-84a2-22e93c2c2b20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:35.260649 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145acb39-df28-46e1-84a2-22e93c2c2b20 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:35.260934 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145acb39-df28-46e1-84a2-22e93c2c2b20 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:35.261255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145acb39-df28-46e1-84a2-22e93c2c2b20 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:35.261999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145acb39-df28-46e1-84a2-22e93c2c2b20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:35.263762 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145acb39-df28-46e1-84a2-22e93c2c2b20 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:35.276747 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1325/2641] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:35.283267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3ffee6-9298-44be-9bcd-9fdc2527e7ed admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:35.297290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3ffee6-9298-44be-9bcd-9fdc2527e7ed admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:35.306320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068ed6d6-05fe-4aad-a4ed-66e9f4c2fd02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:35.307432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068ed6d6-05fe-4aad-a4ed-66e9f4c2fd02 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:35.307895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068ed6d6-05fe-4aad-a4ed-66e9f4c2fd02 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:35.308356 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068ed6d6-05fe-4aad-a4ed-66e9f4c2fd02 admin admin] PATH_INFO: `/v3/projects/3db3f76df57749b3a03fafa4d66197c4/users/1d965c4f51e24da188b990fe3eeba04d/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:35.337366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068ed6d6-05fe-4aad-a4ed-66e9f4c2fd02 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3db3f76df57749b3a03fafa4d66197c4, user_id=1d965c4f51e24da188b990fe3eeba04d, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:35.341231 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068ed6d6-05fe-4aad-a4ed-66e9f4c2fd02 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:35.360085 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1317/2642] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:18:35 2022] PUT /identity/v3/projects/3db3f76df57749b3a03fafa4d66197c4/users/1d965c4f51e24da188b990fe3eeba04d/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:35.368904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145acb39-df28-46e1-84a2-22e93c2c2b20 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:35.382392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145acb39-df28-46e1-84a2-22e93c2c2b20 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:35.459539 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c2c122-dc9e-4b9e-a851-6d024a059729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:35.460268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c2c122-dc9e-4b9e-a851-6d024a059729 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:35.460648 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c2c122-dc9e-4b9e-a851-6d024a059729 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:35.460968 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c2c122-dc9e-4b9e-a851-6d024a059729 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:35.461743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c2c122-dc9e-4b9e-a851-6d024a059729 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:35.463986 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c2c122-dc9e-4b9e-a851-6d024a059729 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:35.488204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1326/2643] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:35 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:35.493731 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068ed6d6-05fe-4aad-a4ed-66e9f4c2fd02 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:35.507067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068ed6d6-05fe-4aad-a4ed-66e9f4c2fd02 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:35.514320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcf577d-bc02-44ff-ac8a-672ab02545b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:35.515122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcf577d-bc02-44ff-ac8a-672ab02545b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:35.515665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcf577d-bc02-44ff-ac8a-672ab02545b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:35.515946 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcf577d-bc02-44ff-ac8a-672ab02545b2 admin admin] PATH_INFO: `/v3/projects/3db3f76df57749b3a03fafa4d66197c4/users/1d965c4f51e24da188b990fe3eeba04d/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:35.518459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcf577d-bc02-44ff-ac8a-672ab02545b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3db3f76df57749b3a03fafa4d66197c4, user_id=1d965c4f51e24da188b990fe3eeba04d, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:35.522509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcf577d-bc02-44ff-ac8a-672ab02545b2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:35.537408 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1318/2644] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:18:35 2022] PUT /identity/v3/projects/3db3f76df57749b3a03fafa4d66197c4/users/1d965c4f51e24da188b990fe3eeba04d/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:18:36.462833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489ce037-a3ac-4b85-a838-9e8f1fcd2670 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:36.463459 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489ce037-a3ac-4b85-a838-9e8f1fcd2670 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:36.464694 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489ce037-a3ac-4b85-a838-9e8f1fcd2670 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:36.507020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-489ce037-a3ac-4b85-a838-9e8f1fcd2670 None None] MFA Rules not processed for user `1d965c4f51e24da188b990fe3eeba04d`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:36.508401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-489ce037-a3ac-4b85-a838-9e8f1fcd2670 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:36.579703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:36.614736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1327/2645] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:36 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:37.850878 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcf577d-bc02-44ff-ac8a-672ab02545b2 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:37.853533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3fcf577d-bc02-44ff-ac8a-672ab02545b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:37.925564 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489ce037-a3ac-4b85-a838-9e8f1fcd2670 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:37.973154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcf577d-bc02-44ff-ac8a-672ab02545b2 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:37.982105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42237c0a-c4dc-44b3-a45f-0fd1603df5ad service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:37.983167 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42237c0a-c4dc-44b3-a45f-0fd1603df5ad service placement] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:37.983740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42237c0a-c4dc-44b3-a45f-0fd1603df5ad service placement] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:37.984202 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42237c0a-c4dc-44b3-a45f-0fd1603df5ad service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:37.987962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-42237c0a-c4dc-44b3-a45f-0fd1603df5ad service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:38.009356 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42237c0a-c4dc-44b3-a45f-0fd1603df5ad service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:38.013564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42237c0a-c4dc-44b3-a45f-0fd1603df5ad service placement] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:38.022817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489ce037-a3ac-4b85-a838-9e8f1fcd2670 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:38.028973 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:38.029635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:38.029932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:38.030180 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] PATH_INFO: `/v3/users/066c282e72aa453fb05eb23b80c1fa11` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:38.031473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] RBAC: Authorizing `identity:delete_user(user_id=066c282e72aa453fb05eb23b80c1fa11)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:38.033447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:38.034859 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1319/2646] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:37 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:18:38.104104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '066c282e72aa453fb05eb23b80c1fa11'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.112164 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '066c282e72aa453fb05eb23b80c1fa11'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.113250 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '066c282e72aa453fb05eb23b80c1fa11'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.136650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '066c282e72aa453fb05eb23b80c1fa11'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.137407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '066c282e72aa453fb05eb23b80c1fa11'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.155235 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1328/2647] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:18:37 2022] DELETE /identity/v3/users/066c282e72aa453fb05eb23b80c1fa11 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:18:38.458176 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42237c0a-c4dc-44b3-a45f-0fd1603df5ad service placement] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:38.475686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42237c0a-c4dc-44b3-a45f-0fd1603df5ad service placement] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:38.538741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:38.579152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:38.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:38.580097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:38.580365 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] PATH_INFO: `/v3/projects/172e4106afc245e3aea7743850e12de3` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:38.581687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] RBAC: Authorizing `identity:delete_project(project_id=172e4106afc245e3aea7743850e12de3)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:38.583631 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:38.650542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '172e4106afc245e3aea7743850e12de3', 'request_id': 'req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.660944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '172e4106afc245e3aea7743850e12de3', 'request_id': 'req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.674464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '172e4106afc245e3aea7743850e12de3', 'request_id': 'req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.701084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] The token cache is being invalidate because project 172e4106afc245e3aea7743850e12de3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:18:38.701713 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.702812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.705589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1320/2648] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:18:38 2022] DELETE /identity/v3/projects/172e4106afc245e3aea7743850e12de3 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:38.710874 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:38.712655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:38.734233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d825b6bf-2f9a-4fdd-89eb-b5cfce6d471f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:38.834193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89799a59-4578-4ed4-be14-defa4de06ced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:38.835293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89799a59-4578-4ed4-be14-defa4de06ced admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:38.835789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89799a59-4578-4ed4-be14-defa4de06ced admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:38.836464 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89799a59-4578-4ed4-be14-defa4de06ced admin admin] PATH_INFO: `/v3/users/1d965c4f51e24da188b990fe3eeba04d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:38.838820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89799a59-4578-4ed4-be14-defa4de06ced admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d965c4f51e24da188b990fe3eeba04d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:38.842556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89799a59-4578-4ed4-be14-defa4de06ced admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:38.856172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:38.942358 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-89799a59-4578-4ed4-be14-defa4de06ced admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d965c4f51e24da188b990fe3eeba04d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.952781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-89799a59-4578-4ed4-be14-defa4de06ced admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d965c4f51e24da188b990fe3eeba04d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.953570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-89799a59-4578-4ed4-be14-defa4de06ced admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d965c4f51e24da188b990fe3eeba04d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.975321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-89799a59-4578-4ed4-be14-defa4de06ced admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d965c4f51e24da188b990fe3eeba04d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.976310 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-89799a59-4578-4ed4-be14-defa4de06ced admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d965c4f51e24da188b990fe3eeba04d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:38.993836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1329/2649] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:18:38 2022] DELETE /identity/v3/users/1d965c4f51e24da188b990fe3eeba04d => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:39.288359 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:39.291292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:39.331227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2127e1-e71b-4bc5-9bfa-8d7fbed40d0e admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:39.423926 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57146ae1-170d-49db-bec3-a4559bf9317c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:39.424588 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57146ae1-170d-49db-bec3-a4559bf9317c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:39.424863 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57146ae1-170d-49db-bec3-a4559bf9317c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:39.425132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57146ae1-170d-49db-bec3-a4559bf9317c admin admin] PATH_INFO: `/v3/projects/3db3f76df57749b3a03fafa4d66197c4` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:39.426411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57146ae1-170d-49db-bec3-a4559bf9317c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3db3f76df57749b3a03fafa4d66197c4)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:39.428412 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57146ae1-170d-49db-bec3-a4559bf9317c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:39.484673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-57146ae1-170d-49db-bec3-a4559bf9317c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3db3f76df57749b3a03fafa4d66197c4', 'request_id': 'req-57146ae1-170d-49db-bec3-a4559bf9317c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:39.492944 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-57146ae1-170d-49db-bec3-a4559bf9317c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3db3f76df57749b3a03fafa4d66197c4', 'request_id': 'req-57146ae1-170d-49db-bec3-a4559bf9317c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:39.501195 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-57146ae1-170d-49db-bec3-a4559bf9317c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3db3f76df57749b3a03fafa4d66197c4', 'request_id': 'req-57146ae1-170d-49db-bec3-a4559bf9317c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:39.508423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:39.520152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-57146ae1-170d-49db-bec3-a4559bf9317c admin admin] The token cache is being invalidate because project 3db3f76df57749b3a03fafa4d66197c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:18:39.520791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-57146ae1-170d-49db-bec3-a4559bf9317c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:39.522567 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-57146ae1-170d-49db-bec3-a4559bf9317c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:39.528852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1321/2650] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:18:39 2022] DELETE /identity/v3/projects/3db3f76df57749b3a03fafa4d66197c4 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:39.530850 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662d4482-0e83-484f-8bdb-7d05d44b6e5c None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:39.531067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662d4482-0e83-484f-8bdb-7d05d44b6e5c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:39.531317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662d4482-0e83-484f-8bdb-7d05d44b6e5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:39.554658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-662d4482-0e83-484f-8bdb-7d05d44b6e5c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:39.556176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-662d4482-0e83-484f-8bdb-7d05d44b6e5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:39.627865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1330/2651] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:39.637590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c67235-993d-4f45-bed4-7b4a804f4495 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:39.638046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c67235-993d-4f45-bed4-7b4a804f4495 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:39.638494 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c67235-993d-4f45-bed4-7b4a804f4495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:39.665879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-45c67235-993d-4f45-bed4-7b4a804f4495 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:39.667175 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-45c67235-993d-4f45-bed4-7b4a804f4495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:39.675760 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1322/2652] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:39.680684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662d4482-0e83-484f-8bdb-7d05d44b6e5c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:39.692133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662d4482-0e83-484f-8bdb-7d05d44b6e5c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:39.697401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a471e2-3a7c-42c6-9b1b-15f5745d9df2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:39.698077 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a471e2-3a7c-42c6-9b1b-15f5745d9df2 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:39.698891 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a471e2-3a7c-42c6-9b1b-15f5745d9df2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:39.699168 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a471e2-3a7c-42c6-9b1b-15f5745d9df2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:39.699969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a471e2-3a7c-42c6-9b1b-15f5745d9df2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:18:39.700274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a471e2-3a7c-42c6-9b1b-15f5745d9df2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:39.701949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a471e2-3a7c-42c6-9b1b-15f5745d9df2 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:39.715748 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1331/2653] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:18:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:18:39.720934 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c67235-993d-4f45-bed4-7b4a804f4495 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:39.731271 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c67235-993d-4f45-bed4-7b4a804f4495 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:39.736615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab53f1f9-aa5a-4e63-8da5-3d574eb0e072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:39.737385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab53f1f9-aa5a-4e63-8da5-3d574eb0e072 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:39.737669 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab53f1f9-aa5a-4e63-8da5-3d574eb0e072 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:39.737932 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab53f1f9-aa5a-4e63-8da5-3d574eb0e072 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:39.738769 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab53f1f9-aa5a-4e63-8da5-3d574eb0e072 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:39.740707 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab53f1f9-aa5a-4e63-8da5-3d574eb0e072 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:39.775072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1323/2654] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:18:39 2022] POST /identity/v3/projects => generated 363 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:18:39.780437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a471e2-3a7c-42c6-9b1b-15f5745d9df2 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:39.791899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a471e2-3a7c-42c6-9b1b-15f5745d9df2 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:39.869300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97eeaf5a-4651-48a8-9ecf-899ed6b9fe09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:39.869930 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eeaf5a-4651-48a8-9ecf-899ed6b9fe09 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:39.870754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eeaf5a-4651-48a8-9ecf-899ed6b9fe09 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:39.871027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eeaf5a-4651-48a8-9ecf-899ed6b9fe09 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:39.871752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97eeaf5a-4651-48a8-9ecf-899ed6b9fe09 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:39.873766 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97eeaf5a-4651-48a8-9ecf-899ed6b9fe09 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:39.905139 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1332/2655] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:18:39 2022] POST /identity/v3/users => generated 282 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:18:39.910282 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab53f1f9-aa5a-4e63-8da5-3d574eb0e072 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:39.930506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab53f1f9-aa5a-4e63-8da5-3d574eb0e072 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:39.939331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762a371f-80eb-46b8-9010-abdb570a12a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:39.940917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762a371f-80eb-46b8-9010-abdb570a12a0 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:39.941403 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762a371f-80eb-46b8-9010-abdb570a12a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:39.941883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762a371f-80eb-46b8-9010-abdb570a12a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:39.943197 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762a371f-80eb-46b8-9010-abdb570a12a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:39.946276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762a371f-80eb-46b8-9010-abdb570a12a0 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:39.961209 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1324/2656] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:39 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:39.966518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97eeaf5a-4651-48a8-9ecf-899ed6b9fe09 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:39.976286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:39.984071 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97eeaf5a-4651-48a8-9ecf-899ed6b9fe09 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:39.990909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eedfeef-3d3e-43e3-9c1f-3d0b79d9e30e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:39.991582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eedfeef-3d3e-43e3-9c1f-3d0b79d9e30e admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:39.991882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eedfeef-3d3e-43e3-9c1f-3d0b79d9e30e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:39.992152 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eedfeef-3d3e-43e3-9c1f-3d0b79d9e30e admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:39.992855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eedfeef-3d3e-43e3-9c1f-3d0b79d9e30e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:39.994518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eedfeef-3d3e-43e3-9c1f-3d0b79d9e30e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:40.009339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1333/2657] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:39 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:40.014995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762a371f-80eb-46b8-9010-abdb570a12a0 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:40.026802 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762a371f-80eb-46b8-9010-abdb570a12a0 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:40.032071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42030ddd-ee59-4619-8c14-3d9df26fbc23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:40.032931 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42030ddd-ee59-4619-8c14-3d9df26fbc23 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:40.033256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42030ddd-ee59-4619-8c14-3d9df26fbc23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:40.033690 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42030ddd-ee59-4619-8c14-3d9df26fbc23 admin admin] PATH_INFO: `/v3/projects/de4db9b6f89a4bde8d46d2dff214f5a0/users/d42b9ff92d0842f095b9c81603c25bbe/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:40.052205 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42030ddd-ee59-4619-8c14-3d9df26fbc23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de4db9b6f89a4bde8d46d2dff214f5a0, user_id=d42b9ff92d0842f095b9c81603c25bbe, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:40.055513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42030ddd-ee59-4619-8c14-3d9df26fbc23 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:40.072088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1325/2658] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:18:40 2022] PUT /identity/v3/projects/de4db9b6f89a4bde8d46d2dff214f5a0/users/d42b9ff92d0842f095b9c81603c25bbe/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:18:40.085435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cc925e-249e-40da-b8f3-0501294ea0cf None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:40.085923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cc925e-249e-40da-b8f3-0501294ea0cf None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:40.086381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cc925e-249e-40da-b8f3-0501294ea0cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:40.264665 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-95cc925e-249e-40da-b8f3-0501294ea0cf None None] MFA Rules not processed for user `d42b9ff92d0842f095b9c81603c25bbe`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:40.266794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-95cc925e-249e-40da-b8f3-0501294ea0cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:40.365898 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1334/2659] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:40 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:40.371818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42030ddd-ee59-4619-8c14-3d9df26fbc23 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:40.446888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42030ddd-ee59-4619-8c14-3d9df26fbc23 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:40.452975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f737b22d-94fc-4759-8399-03069f2baec5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:40.453675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f737b22d-94fc-4759-8399-03069f2baec5 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:40.454032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f737b22d-94fc-4759-8399-03069f2baec5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:40.454421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f737b22d-94fc-4759-8399-03069f2baec5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:40.455218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f737b22d-94fc-4759-8399-03069f2baec5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:40.457376 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f737b22d-94fc-4759-8399-03069f2baec5 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:40.490095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1326/2660] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:18:40 2022] POST /identity/v3/projects => generated 365 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:18:40.495600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cc925e-249e-40da-b8f3-0501294ea0cf None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:40.508124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cc925e-249e-40da-b8f3-0501294ea0cf None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:40.576544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82b1ea3-4a05-4f3b-a2dd-5820f194f494 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:40.577195 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82b1ea3-4a05-4f3b-a2dd-5820f194f494 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:40.577473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82b1ea3-4a05-4f3b-a2dd-5820f194f494 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:40.577743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82b1ea3-4a05-4f3b-a2dd-5820f194f494 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:40.578432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82b1ea3-4a05-4f3b-a2dd-5820f194f494 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:40.580540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82b1ea3-4a05-4f3b-a2dd-5820f194f494 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:40.607749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1335/2661] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:18:40 2022] POST /identity/v3/users => generated 289 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:18:40.613537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f737b22d-94fc-4759-8399-03069f2baec5 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:40.625884 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f737b22d-94fc-4759-8399-03069f2baec5 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:40.633718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c4fc73-4380-474c-b703-7b4f084a70d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:40.634772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c4fc73-4380-474c-b703-7b4f084a70d5 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:40.635284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c4fc73-4380-474c-b703-7b4f084a70d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:40.636506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c4fc73-4380-474c-b703-7b4f084a70d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:40.637803 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c4fc73-4380-474c-b703-7b4f084a70d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:40.641121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c4fc73-4380-474c-b703-7b4f084a70d5 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:40.661296 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1327/2662] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:40 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:40.666020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82b1ea3-4a05-4f3b-a2dd-5820f194f494 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:40.677203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82b1ea3-4a05-4f3b-a2dd-5820f194f494 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:40.683166 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba719e7-0308-47e1-9a77-5656d89da1e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:40.683906 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba719e7-0308-47e1-9a77-5656d89da1e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:40.684181 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba719e7-0308-47e1-9a77-5656d89da1e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:40.684452 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba719e7-0308-47e1-9a77-5656d89da1e5 admin admin] PATH_INFO: `/v3/projects/ce138a6c54454aa487ca3ee427e8a391/users/49d6da1b7f4942a6a17365a7dfe2979a/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:40.706193 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba719e7-0308-47e1-9a77-5656d89da1e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce138a6c54454aa487ca3ee427e8a391, user_id=49d6da1b7f4942a6a17365a7dfe2979a, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:40.708972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba719e7-0308-47e1-9a77-5656d89da1e5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:40.725101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1336/2663] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:18:40 2022] PUT /identity/v3/projects/ce138a6c54454aa487ca3ee427e8a391/users/49d6da1b7f4942a6a17365a7dfe2979a/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:40.730043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c4fc73-4380-474c-b703-7b4f084a70d5 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:40.741596 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c4fc73-4380-474c-b703-7b4f084a70d5 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:40.809899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9232dec7-189a-4cb2-a5b6-39e8aed9c11c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:40.810604 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9232dec7-189a-4cb2-a5b6-39e8aed9c11c admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:40.810919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9232dec7-189a-4cb2-a5b6-39e8aed9c11c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:40.811163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9232dec7-189a-4cb2-a5b6-39e8aed9c11c admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:40.812004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9232dec7-189a-4cb2-a5b6-39e8aed9c11c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:40.813660 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9232dec7-189a-4cb2-a5b6-39e8aed9c11c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:40.820410 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:40.827699 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1328/2664] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:40 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:40.833838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba719e7-0308-47e1-9a77-5656d89da1e5 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:40.849114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba719e7-0308-47e1-9a77-5656d89da1e5 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:40.855882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50976606-2f1d-4589-a603-da02aee7d0e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:40.856578 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50976606-2f1d-4589-a603-da02aee7d0e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:40.856856 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50976606-2f1d-4589-a603-da02aee7d0e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:40.857126 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50976606-2f1d-4589-a603-da02aee7d0e8 admin admin] PATH_INFO: `/v3/projects/ce138a6c54454aa487ca3ee427e8a391/users/49d6da1b7f4942a6a17365a7dfe2979a/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:40.859236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50976606-2f1d-4589-a603-da02aee7d0e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce138a6c54454aa487ca3ee427e8a391, user_id=49d6da1b7f4942a6a17365a7dfe2979a, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:40.861923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50976606-2f1d-4589-a603-da02aee7d0e8 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:40.874287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1337/2665] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:18:40 2022] PUT /identity/v3/projects/ce138a6c54454aa487ca3ee427e8a391/users/49d6da1b7f4942a6a17365a7dfe2979a/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:40.886842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0023ab18-6d1e-4bb9-83d5-fc3e055d25eb None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:40.887163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0023ab18-6d1e-4bb9-83d5-fc3e055d25eb None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:40.887579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0023ab18-6d1e-4bb9-83d5-fc3e055d25eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:40.922508 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-0023ab18-6d1e-4bb9-83d5-fc3e055d25eb None None] MFA Rules not processed for user `49d6da1b7f4942a6a17365a7dfe2979a`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:40.923823 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0023ab18-6d1e-4bb9-83d5-fc3e055d25eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:41.006078 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1329/2666] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:40 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:18:41.018707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50976606-2f1d-4589-a603-da02aee7d0e8 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:41.020551 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-50976606-2f1d-4589-a603-da02aee7d0e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:41.060905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:41.088944 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50976606-2f1d-4589-a603-da02aee7d0e8 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:41.095122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d217243f-0e58-4573-bf90-2f394b7de7e2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:41.096238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d217243f-0e58-4573-bf90-2f394b7de7e2 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:41.096749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d217243f-0e58-4573-bf90-2f394b7de7e2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:41.097254 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d217243f-0e58-4573-bf90-2f394b7de7e2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:41.114475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d217243f-0e58-4573-bf90-2f394b7de7e2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:41.117328 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d217243f-0e58-4573-bf90-2f394b7de7e2 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:41.137503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1338/2667] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:41 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:18:41.226078 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0023ab18-6d1e-4bb9-83d5-fc3e055d25eb None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:41.236467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0023ab18-6d1e-4bb9-83d5-fc3e055d25eb None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:41.240290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ab49e7-5216-4a0d-af5d-47af87c2f192 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:41.240922 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ab49e7-5216-4a0d-af5d-47af87c2f192 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:41.241200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ab49e7-5216-4a0d-af5d-47af87c2f192 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:41.241468 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ab49e7-5216-4a0d-af5d-47af87c2f192 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:41.326493 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ab49e7-5216-4a0d-af5d-47af87c2f192 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:41.328495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ab49e7-5216-4a0d-af5d-47af87c2f192 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:41.339981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1330/2668] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:41 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:18:44.264165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d217243f-0e58-4573-bf90-2f394b7de7e2 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:44.343906 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d217243f-0e58-4573-bf90-2f394b7de7e2 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:44.348866 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:44.349529 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:44.349805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:44.350075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] PATH_INFO: `/v3/users/d42b9ff92d0842f095b9c81603c25bbe` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:44.351284 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d42b9ff92d0842f095b9c81603c25bbe)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:44.353263 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:44.419069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd42b9ff92d0842f095b9c81603c25bbe'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:44.427120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd42b9ff92d0842f095b9c81603c25bbe'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:44.427921 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd42b9ff92d0842f095b9c81603c25bbe'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:44.443055 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd42b9ff92d0842f095b9c81603c25bbe'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:44.444061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd42b9ff92d0842f095b9c81603c25bbe'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:44.460440 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1339/2669] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:18:44 2022] DELETE /identity/v3/users/d42b9ff92d0842f095b9c81603c25bbe => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:44.465856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadf9e43-ac02-46aa-8863-f79def054ed9 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:44.466509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadf9e43-ac02-46aa-8863-f79def054ed9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:44.466509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadf9e43-ac02-46aa-8863-f79def054ed9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:44.487667 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-aadf9e43-ac02-46aa-8863-f79def054ed9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:44.488916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-aadf9e43-ac02-46aa-8863-f79def054ed9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:44.587239 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1331/2670] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:44.598993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:44.600929 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:44.669800 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d5783f-f795-4cd5-add7-2433e8bc2933 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:44.675317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6f3793-237a-40b0-9225-4ce702349090 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:44.676457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6f3793-237a-40b0-9225-4ce702349090 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:44.676931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6f3793-237a-40b0-9225-4ce702349090 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:44.677406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6f3793-237a-40b0-9225-4ce702349090 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:44.695603 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6f3793-237a-40b0-9225-4ce702349090 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:44.698406 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6f3793-237a-40b0-9225-4ce702349090 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:44.706101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:44.718868 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1340/2671] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:18:45.150695 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aadf9e43-ac02-46aa-8863-f79def054ed9 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:45.152420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-aadf9e43-ac02-46aa-8863-f79def054ed9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:45.166705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aadf9e43-ac02-46aa-8863-f79def054ed9 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:45.171704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:45.172439 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:45.172687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:45.172960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] PATH_INFO: `/v3/projects/de4db9b6f89a4bde8d46d2dff214f5a0` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:45.174402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] RBAC: Authorizing `identity:delete_project(project_id=de4db9b6f89a4bde8d46d2dff214f5a0)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:45.177490 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:45.212988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:45.248130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de4db9b6f89a4bde8d46d2dff214f5a0', 'request_id': 'req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:45.256101 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de4db9b6f89a4bde8d46d2dff214f5a0', 'request_id': 'req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:45.267116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de4db9b6f89a4bde8d46d2dff214f5a0', 'request_id': 'req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:45.280794 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] The token cache is being invalidate because project de4db9b6f89a4bde8d46d2dff214f5a0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:18:45.281117 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:45.281875 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:45.283954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1332/2672] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:18:45 2022] DELETE /identity/v3/projects/de4db9b6f89a4bde8d46d2dff214f5a0 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:45.288997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6f3793-237a-40b0-9225-4ce702349090 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:45.299878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6f3793-237a-40b0-9225-4ce702349090 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:45.372408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d27dd21-328b-4499-9ab3-7f23e4f6eab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:45.373357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d27dd21-328b-4499-9ab3-7f23e4f6eab3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:45.373842 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d27dd21-328b-4499-9ab3-7f23e4f6eab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:45.374315 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d27dd21-328b-4499-9ab3-7f23e4f6eab3 admin admin] PATH_INFO: `/v3/users/49d6da1b7f4942a6a17365a7dfe2979a` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:45.375938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d27dd21-328b-4499-9ab3-7f23e4f6eab3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=49d6da1b7f4942a6a17365a7dfe2979a)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:45.378179 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d27dd21-328b-4499-9ab3-7f23e4f6eab3 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:45.445937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4d27dd21-328b-4499-9ab3-7f23e4f6eab3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '49d6da1b7f4942a6a17365a7dfe2979a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:45.459558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4d27dd21-328b-4499-9ab3-7f23e4f6eab3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '49d6da1b7f4942a6a17365a7dfe2979a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:45.461011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4d27dd21-328b-4499-9ab3-7f23e4f6eab3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '49d6da1b7f4942a6a17365a7dfe2979a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:45.484474 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4d27dd21-328b-4499-9ab3-7f23e4f6eab3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '49d6da1b7f4942a6a17365a7dfe2979a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:45.485712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4d27dd21-328b-4499-9ab3-7f23e4f6eab3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '49d6da1b7f4942a6a17365a7dfe2979a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:45.509278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1341/2673] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:18:45 2022] DELETE /identity/v3/users/49d6da1b7f4942a6a17365a7dfe2979a => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:45.928063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:45.929801 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:45.947915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8219cf-bc17-41d8-ba63-bdbc049b6d8c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:46.048989 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:46.049811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:46.050181 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:46.050582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee admin admin] PATH_INFO: `/v3/projects/ce138a6c54454aa487ca3ee427e8a391` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:46.052143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce138a6c54454aa487ca3ee427e8a391)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:46.055115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:46.128988 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce138a6c54454aa487ca3ee427e8a391', 'request_id': 'req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:46.138039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce138a6c54454aa487ca3ee427e8a391', 'request_id': 'req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:46.149385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce138a6c54454aa487ca3ee427e8a391', 'request_id': 'req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:46.167843 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee admin admin] The token cache is being invalidate because project ce138a6c54454aa487ca3ee427e8a391 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:18:46.168196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:46.169252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-07aee18f-30eb-4b57-9e5e-c72ea65b8bee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:46.173394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1333/2674] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:18:45 2022] DELETE /identity/v3/projects/ce138a6c54454aa487ca3ee427e8a391 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:46.189019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916b73b9-44e7-48d4-96c7-366b5f43d080 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:46.189438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916b73b9-44e7-48d4-96c7-366b5f43d080 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:46.189823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916b73b9-44e7-48d4-96c7-366b5f43d080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:46.216356 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-916b73b9-44e7-48d4-96c7-366b5f43d080 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:46.217605 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-916b73b9-44e7-48d4-96c7-366b5f43d080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:46.286671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1342/2675] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:46.295675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a908d4-819b-4f50-bbbc-7051aa49d76f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:46.296012 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a908d4-819b-4f50-bbbc-7051aa49d76f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:46.296266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a908d4-819b-4f50-bbbc-7051aa49d76f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:46.320216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-91a908d4-819b-4f50-bbbc-7051aa49d76f None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:46.322324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-91a908d4-819b-4f50-bbbc-7051aa49d76f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:46.337129 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1334/2676] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:18:46.344597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916b73b9-44e7-48d4-96c7-366b5f43d080 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:46.362619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916b73b9-44e7-48d4-96c7-366b5f43d080 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:46.370846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921e4bbc-70f9-47e8-8ff1-3098263e4efa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:46.371956 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921e4bbc-70f9-47e8-8ff1-3098263e4efa admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:46.372449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921e4bbc-70f9-47e8-8ff1-3098263e4efa admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:46.372926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921e4bbc-70f9-47e8-8ff1-3098263e4efa admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:46.374171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921e4bbc-70f9-47e8-8ff1-3098263e4efa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:18:46.374654 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921e4bbc-70f9-47e8-8ff1-3098263e4efa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:46.377684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921e4bbc-70f9-47e8-8ff1-3098263e4efa admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:46.399802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1343/2677] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:18:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:18:46.404710 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a908d4-819b-4f50-bbbc-7051aa49d76f None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:46.414840 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a908d4-819b-4f50-bbbc-7051aa49d76f None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:46.420543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fe50d0-cf06-4642-8286-302a60b78cc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:46.421185 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fe50d0-cf06-4642-8286-302a60b78cc2 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:46.421465 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fe50d0-cf06-4642-8286-302a60b78cc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:46.421744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fe50d0-cf06-4642-8286-302a60b78cc2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:46.422463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fe50d0-cf06-4642-8286-302a60b78cc2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:46.424561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fe50d0-cf06-4642-8286-302a60b78cc2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:46.457209 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1335/2678] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:18:46 2022] POST /identity/v3/projects => generated 387 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:18:46.461595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921e4bbc-70f9-47e8-8ff1-3098263e4efa admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:46.469823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:46.478770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921e4bbc-70f9-47e8-8ff1-3098263e4efa admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:46.558030 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe57d333-f0f2-4088-88ef-e6a6937a7392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:46.558667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe57d333-f0f2-4088-88ef-e6a6937a7392 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:46.558948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe57d333-f0f2-4088-88ef-e6a6937a7392 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:46.559222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe57d333-f0f2-4088-88ef-e6a6937a7392 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:46.559951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe57d333-f0f2-4088-88ef-e6a6937a7392 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:46.561924 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe57d333-f0f2-4088-88ef-e6a6937a7392 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:46.586300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1344/2679] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:18:46 2022] POST /identity/v3/users => generated 294 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:18:46.591478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fe50d0-cf06-4642-8286-302a60b78cc2 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:46.604265 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fe50d0-cf06-4642-8286-302a60b78cc2 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:46.610728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d4f5ac-d7c3-486e-9134-be72cd9de646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:46.611627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d4f5ac-d7c3-486e-9134-be72cd9de646 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:46.612017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d4f5ac-d7c3-486e-9134-be72cd9de646 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:46.612423 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d4f5ac-d7c3-486e-9134-be72cd9de646 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:46.613256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d4f5ac-d7c3-486e-9134-be72cd9de646 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:46.615701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d4f5ac-d7c3-486e-9134-be72cd9de646 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:46.618586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:46.630936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1336/2680] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:46 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:46.636238 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe57d333-f0f2-4088-88ef-e6a6937a7392 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:46.647551 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe57d333-f0f2-4088-88ef-e6a6937a7392 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:46.653182 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da9b6cd-3283-42bc-b8cd-0bd349bf4356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:46.653974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da9b6cd-3283-42bc-b8cd-0bd349bf4356 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:46.654360 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da9b6cd-3283-42bc-b8cd-0bd349bf4356 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:46.654797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da9b6cd-3283-42bc-b8cd-0bd349bf4356 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:46.655775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da9b6cd-3283-42bc-b8cd-0bd349bf4356 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:46.657538 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da9b6cd-3283-42bc-b8cd-0bd349bf4356 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:46.670774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1345/2681] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:46 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:18:46.676190 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d4f5ac-d7c3-486e-9134-be72cd9de646 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:46.693705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d4f5ac-d7c3-486e-9134-be72cd9de646 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:46.701048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f428e6f3-80a5-43e2-a7b5-32d30ff4a401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:46.702136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f428e6f3-80a5-43e2-a7b5-32d30ff4a401 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:46.702600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f428e6f3-80a5-43e2-a7b5-32d30ff4a401 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:46.703057 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f428e6f3-80a5-43e2-a7b5-32d30ff4a401 admin admin] PATH_INFO: `/v3/projects/d8a704d95de14640bd14391535726ef4/users/33699fb6c1ac43c5b5eaba575e2d35b5/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:46.729498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f428e6f3-80a5-43e2-a7b5-32d30ff4a401 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d8a704d95de14640bd14391535726ef4, user_id=33699fb6c1ac43c5b5eaba575e2d35b5, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:46.733616 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f428e6f3-80a5-43e2-a7b5-32d30ff4a401 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:46.751811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1337/2682] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:18:46 2022] PUT /identity/v3/projects/d8a704d95de14640bd14391535726ef4/users/33699fb6c1ac43c5b5eaba575e2d35b5/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:18:46.766575 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea4a437-1320-4fa9-a8a0-a52beed6b082 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:46.767000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea4a437-1320-4fa9-a8a0-a52beed6b082 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:46.767409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea4a437-1320-4fa9-a8a0-a52beed6b082 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:46.816729 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-dea4a437-1320-4fa9-a8a0-a52beed6b082 None None] MFA Rules not processed for user `33699fb6c1ac43c5b5eaba575e2d35b5`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:46.818048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-dea4a437-1320-4fa9-a8a0-a52beed6b082 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:46.895558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1346/2683] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:46 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:18:46.902270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f428e6f3-80a5-43e2-a7b5-32d30ff4a401 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:46.992473 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f428e6f3-80a5-43e2-a7b5-32d30ff4a401 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:47.001088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1363b355-0b8a-4e94-a371-a4cde450f8ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:47.002186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1363b355-0b8a-4e94-a371-a4cde450f8ee admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:47.002664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1363b355-0b8a-4e94-a371-a4cde450f8ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:47.003153 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1363b355-0b8a-4e94-a371-a4cde450f8ee admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:47.004664 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1363b355-0b8a-4e94-a371-a4cde450f8ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:47.008250 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1363b355-0b8a-4e94-a371-a4cde450f8ee admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:47.059573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1338/2684] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:18:46 2022] POST /identity/v3/projects => generated 385 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:18:47.064792 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea4a437-1320-4fa9-a8a0-a52beed6b082 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:47.078359 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea4a437-1320-4fa9-a8a0-a52beed6b082 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:47.189455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56389128-f327-4bef-b845-efde0f3522cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:47.190225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56389128-f327-4bef-b845-efde0f3522cd admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:47.190831 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56389128-f327-4bef-b845-efde0f3522cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:47.191344 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56389128-f327-4bef-b845-efde0f3522cd admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:47.192455 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56389128-f327-4bef-b845-efde0f3522cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:47.195044 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56389128-f327-4bef-b845-efde0f3522cd admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:47.224676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1347/2685] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:18:47 2022] POST /identity/v3/users => generated 299 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:18:47.232174 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1363b355-0b8a-4e94-a371-a4cde450f8ee admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:47.245640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1363b355-0b8a-4e94-a371-a4cde450f8ee admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:47.251785 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ba896f-d28e-4ef6-9244-8e2d3c5d2029 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:47.252424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ba896f-d28e-4ef6-9244-8e2d3c5d2029 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:47.252703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ba896f-d28e-4ef6-9244-8e2d3c5d2029 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:47.252973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ba896f-d28e-4ef6-9244-8e2d3c5d2029 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:47.253682 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ba896f-d28e-4ef6-9244-8e2d3c5d2029 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:47.255892 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ba896f-d28e-4ef6-9244-8e2d3c5d2029 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:47.269967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1339/2686] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:47 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:47.275443 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56389128-f327-4bef-b845-efde0f3522cd admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:47.285133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:47.296153 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56389128-f327-4bef-b845-efde0f3522cd admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:47.303180 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff40ba82-840b-4705-90d4-0296d2c36ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:47.304013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff40ba82-840b-4705-90d4-0296d2c36ba1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:47.304292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff40ba82-840b-4705-90d4-0296d2c36ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:47.304563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff40ba82-840b-4705-90d4-0296d2c36ba1 admin admin] PATH_INFO: `/v3/projects/3984cd052c3a435cbc3a8413d489808e/users/1d2b3591cb864c2e92711730c3ff2363/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:47.323732 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff40ba82-840b-4705-90d4-0296d2c36ba1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3984cd052c3a435cbc3a8413d489808e, user_id=1d2b3591cb864c2e92711730c3ff2363, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:47.327043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff40ba82-840b-4705-90d4-0296d2c36ba1 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:47.345322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1348/2687] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:18:47 2022] PUT /identity/v3/projects/3984cd052c3a435cbc3a8413d489808e/users/1d2b3591cb864c2e92711730c3ff2363/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:47.350172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ba896f-d28e-4ef6-9244-8e2d3c5d2029 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:47.362435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ba896f-d28e-4ef6-9244-8e2d3c5d2029 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:47.438676 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b8f777-0575-4e44-b600-e1f1ccef7929 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:47.439796 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b8f777-0575-4e44-b600-e1f1ccef7929 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:47.440228 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b8f777-0575-4e44-b600-e1f1ccef7929 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:47.440634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b8f777-0575-4e44-b600-e1f1ccef7929 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:47.441699 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b8f777-0575-4e44-b600-e1f1ccef7929 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:47.444668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b8f777-0575-4e44-b600-e1f1ccef7929 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:47.464826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1340/2688] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:47 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:47.469436 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff40ba82-840b-4705-90d4-0296d2c36ba1 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:47.482021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff40ba82-840b-4705-90d4-0296d2c36ba1 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:47.487302 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c91c52-1653-47b9-96b9-efe476f8ab3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:47.488222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c91c52-1653-47b9-96b9-efe476f8ab3f admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:47.488576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c91c52-1653-47b9-96b9-efe476f8ab3f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:47.488997 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c91c52-1653-47b9-96b9-efe476f8ab3f admin admin] PATH_INFO: `/v3/projects/3984cd052c3a435cbc3a8413d489808e/users/1d2b3591cb864c2e92711730c3ff2363/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:47.491838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c91c52-1653-47b9-96b9-efe476f8ab3f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3984cd052c3a435cbc3a8413d489808e, user_id=1d2b3591cb864c2e92711730c3ff2363, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:47.495475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c91c52-1653-47b9-96b9-efe476f8ab3f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:47.510509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1349/2689] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:18:47 2022] PUT /identity/v3/projects/3984cd052c3a435cbc3a8413d489808e/users/1d2b3591cb864c2e92711730c3ff2363/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:18:47.521424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a844cb17-06a9-4ff1-aa7d-57c6903306fa None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:47.521708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a844cb17-06a9-4ff1-aa7d-57c6903306fa None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:47.521983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a844cb17-06a9-4ff1-aa7d-57c6903306fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:47.559529 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a844cb17-06a9-4ff1-aa7d-57c6903306fa None None] MFA Rules not processed for user `1d2b3591cb864c2e92711730c3ff2363`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:47.561085 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a844cb17-06a9-4ff1-aa7d-57c6903306fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:47.602385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:47.649483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1341/2690] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:47 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:47.665448 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c91c52-1653-47b9-96b9-efe476f8ab3f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:47.667502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-07c91c52-1653-47b9-96b9-efe476f8ab3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:47.736409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c91c52-1653-47b9-96b9-efe476f8ab3f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:47.742279 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d8ca63-cef1-403b-9aa2-7d13a3b91072 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:47.743486 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d8ca63-cef1-403b-9aa2-7d13a3b91072 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:47.743961 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d8ca63-cef1-403b-9aa2-7d13a3b91072 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:47.744441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d8ca63-cef1-403b-9aa2-7d13a3b91072 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:47.756696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d8ca63-cef1-403b-9aa2-7d13a3b91072 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:47.758457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d8ca63-cef1-403b-9aa2-7d13a3b91072 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:47.769865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1350/2691] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:47 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:18:47.844028 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a844cb17-06a9-4ff1-aa7d-57c6903306fa None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:47.943473 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a844cb17-06a9-4ff1-aa7d-57c6903306fa None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:47.949982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:47.950995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:47.951279 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:47.951654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] PATH_INFO: `/v3/users/33699fb6c1ac43c5b5eaba575e2d35b5` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:47.953131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] RBAC: Authorizing `identity:delete_user(user_id=33699fb6c1ac43c5b5eaba575e2d35b5)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:47.955256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:48.021898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '33699fb6c1ac43c5b5eaba575e2d35b5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:48.022780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '33699fb6c1ac43c5b5eaba575e2d35b5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:48.032976 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '33699fb6c1ac43c5b5eaba575e2d35b5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:48.049031 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '33699fb6c1ac43c5b5eaba575e2d35b5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:48.061920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '33699fb6c1ac43c5b5eaba575e2d35b5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:48.067463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1342/2692] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:18:47 2022] DELETE /identity/v3/users/33699fb6c1ac43c5b5eaba575e2d35b5 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:18:48.073080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ee9e7e-bb7c-41bf-a367-5a8c0f531b0c None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:48.074354 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ee9e7e-bb7c-41bf-a367-5a8c0f531b0c None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:48.074956 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ee9e7e-bb7c-41bf-a367-5a8c0f531b0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:48.106491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-99ee9e7e-bb7c-41bf-a367-5a8c0f531b0c None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:48.108473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-99ee9e7e-bb7c-41bf-a367-5a8c0f531b0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:48.220460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1351/2693] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:48.233149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:48.234997 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:48.303127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:48.309360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b030f68a-a612-4a69-83e8-25d9bd95ac45 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:48.315722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab620ef0-e1af-4b43-b6e5-27808a2748ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:48.316672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab620ef0-e1af-4b43-b6e5-27808a2748ab service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:48.317155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab620ef0-e1af-4b43-b6e5-27808a2748ab service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:48.317500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab620ef0-e1af-4b43-b6e5-27808a2748ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:48.330501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab620ef0-e1af-4b43-b6e5-27808a2748ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:48.332104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab620ef0-e1af-4b43-b6e5-27808a2748ab service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:48.345123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1343/2694] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:18:48.693314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ee9e7e-bb7c-41bf-a367-5a8c0f531b0c None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:48.696038 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-99ee9e7e-bb7c-41bf-a367-5a8c0f531b0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:48.715993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ee9e7e-bb7c-41bf-a367-5a8c0f531b0c None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:48.724258 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:48.725387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:48.725881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:48.726357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] PATH_INFO: `/v3/projects/d8a704d95de14640bd14391535726ef4` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:48.728712 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d8a704d95de14640bd14391535726ef4)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:48.731656 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:48.814781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd8a704d95de14640bd14391535726ef4', 'request_id': 'req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:48.827713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd8a704d95de14640bd14391535726ef4', 'request_id': 'req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:48.837576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd8a704d95de14640bd14391535726ef4', 'request_id': 'req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:48.841293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:48.862104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] The token cache is being invalidate because project d8a704d95de14640bd14391535726ef4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:18:48.862470 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:48.863140 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:48.864790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1352/2695] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:18:48 2022] DELETE /identity/v3/projects/d8a704d95de14640bd14391535726ef4 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:48.870800 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab620ef0-e1af-4b43-b6e5-27808a2748ab service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:48.882429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab620ef0-e1af-4b43-b6e5-27808a2748ab service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:48.957797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997c521e-2a45-43fc-bdab-e42019183e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:48.958454 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997c521e-2a45-43fc-bdab-e42019183e75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:48.958735 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997c521e-2a45-43fc-bdab-e42019183e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:48.959007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997c521e-2a45-43fc-bdab-e42019183e75 admin admin] PATH_INFO: `/v3/users/1d2b3591cb864c2e92711730c3ff2363` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:48.960311 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997c521e-2a45-43fc-bdab-e42019183e75 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d2b3591cb864c2e92711730c3ff2363)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:48.962292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997c521e-2a45-43fc-bdab-e42019183e75 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:49.021490 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-997c521e-2a45-43fc-bdab-e42019183e75 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d2b3591cb864c2e92711730c3ff2363'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:49.022391 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-997c521e-2a45-43fc-bdab-e42019183e75 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d2b3591cb864c2e92711730c3ff2363'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:49.040320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-997c521e-2a45-43fc-bdab-e42019183e75 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d2b3591cb864c2e92711730c3ff2363'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:49.061222 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-997c521e-2a45-43fc-bdab-e42019183e75 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d2b3591cb864c2e92711730c3ff2363'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:49.070043 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-997c521e-2a45-43fc-bdab-e42019183e75 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d2b3591cb864c2e92711730c3ff2363'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:49.077395 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1344/2696] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:18:48 2022] DELETE /identity/v3/users/1d2b3591cb864c2e92711730c3ff2363 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:18:49.451493 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:49.452982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:49.468650 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfb7285-c8a2-4d09-8b7f-6d39d32530a7 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:49.554457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:49.555247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:49.555859 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:49.556271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a admin admin] PATH_INFO: `/v3/projects/3984cd052c3a435cbc3a8413d489808e` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:49.558606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a admin admin] RBAC: Authorizing `identity:delete_project(project_id=3984cd052c3a435cbc3a8413d489808e)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:49.560746 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:49.619098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3984cd052c3a435cbc3a8413d489808e', 'request_id': 'req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:49.627428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3984cd052c3a435cbc3a8413d489808e', 'request_id': 'req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:49.634484 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3984cd052c3a435cbc3a8413d489808e', 'request_id': 'req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:49.651143 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a admin admin] The token cache is being invalidate because project 3984cd052c3a435cbc3a8413d489808e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:18:49.651885 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:49.653088 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-091def1d-82dc-4bbb-bfde-0b1b5bf2b52a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:49.654896 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1353/2697] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:18:49 2022] DELETE /identity/v3/projects/3984cd052c3a435cbc3a8413d489808e => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:49.662307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837923f5-a7d4-445a-9e4d-8594540f4169 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:49.662748 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837923f5-a7d4-445a-9e4d-8594540f4169 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:49.663151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837923f5-a7d4-445a-9e4d-8594540f4169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:49.686408 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-837923f5-a7d4-445a-9e4d-8594540f4169 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:49.687681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-837923f5-a7d4-445a-9e4d-8594540f4169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:49.767149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1345/2698] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:49.776509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf795c50-0251-497f-915b-38dcc70c4b08 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:49.777479 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf795c50-0251-497f-915b-38dcc70c4b08 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:49.777479 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf795c50-0251-497f-915b-38dcc70c4b08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:49.798691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-bf795c50-0251-497f-915b-38dcc70c4b08 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:49.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-bf795c50-0251-497f-915b-38dcc70c4b08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:49.809008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1354/2699] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:49.813641 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837923f5-a7d4-445a-9e4d-8594540f4169 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:49.824856 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837923f5-a7d4-445a-9e4d-8594540f4169 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:49.830046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c375732c-0369-428a-958c-44ba33cafc57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:49.830700 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c375732c-0369-428a-958c-44ba33cafc57 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:49.830982 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c375732c-0369-428a-958c-44ba33cafc57 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:49.831253 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c375732c-0369-428a-958c-44ba33cafc57 admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:49.832010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c375732c-0369-428a-958c-44ba33cafc57 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:18:49.832315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c375732c-0369-428a-958c-44ba33cafc57 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:49.833952 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c375732c-0369-428a-958c-44ba33cafc57 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:49.841870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:49.849088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1346/2700] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:18:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:18:49.854109 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf795c50-0251-497f-915b-38dcc70c4b08 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:49.870858 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf795c50-0251-497f-915b-38dcc70c4b08 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:49.878846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea38d48a-1140-491d-aebe-51a613e0a450 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:49.879933 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea38d48a-1140-491d-aebe-51a613e0a450 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:49.880477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea38d48a-1140-491d-aebe-51a613e0a450 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:49.880963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea38d48a-1140-491d-aebe-51a613e0a450 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:49.882159 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea38d48a-1140-491d-aebe-51a613e0a450 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:49.885989 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea38d48a-1140-491d-aebe-51a613e0a450 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:49.922246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1355/2701] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:18:49 2022] POST /identity/v3/projects => generated 383 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:18:49.927590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c375732c-0369-428a-958c-44ba33cafc57 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:49.939886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c375732c-0369-428a-958c-44ba33cafc57 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:50.016155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea041d0-2763-4fad-81e1-f5502310c15a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:50.016834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea041d0-2763-4fad-81e1-f5502310c15a admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:50.017097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea041d0-2763-4fad-81e1-f5502310c15a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:50.017481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea041d0-2763-4fad-81e1-f5502310c15a admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:50.018366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea041d0-2763-4fad-81e1-f5502310c15a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:50.020501 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea041d0-2763-4fad-81e1-f5502310c15a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:50.044973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1347/2702] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:18:49 2022] POST /identity/v3/users => generated 292 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:18:50.049769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea38d48a-1140-491d-aebe-51a613e0a450 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:50.062229 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea38d48a-1140-491d-aebe-51a613e0a450 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:50.067413 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e867fb-4a1f-45cd-aa93-0454c25842bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:50.068494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e867fb-4a1f-45cd-aa93-0454c25842bd admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:50.068841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e867fb-4a1f-45cd-aa93-0454c25842bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:50.069176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e867fb-4a1f-45cd-aa93-0454c25842bd admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:50.069991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e867fb-4a1f-45cd-aa93-0454c25842bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:50.072130 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e867fb-4a1f-45cd-aa93-0454c25842bd admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:50.084854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1356/2703] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:50 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:50.089366 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea041d0-2763-4fad-81e1-f5502310c15a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:50.100919 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea041d0-2763-4fad-81e1-f5502310c15a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:50.107316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf37da3-dd3d-4e2c-b2dd-d65050a9d446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:50.108048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf37da3-dd3d-4e2c-b2dd-d65050a9d446 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:50.108386 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf37da3-dd3d-4e2c-b2dd-d65050a9d446 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:50.108707 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf37da3-dd3d-4e2c-b2dd-d65050a9d446 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:50.109471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf37da3-dd3d-4e2c-b2dd-d65050a9d446 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:50.111290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf37da3-dd3d-4e2c-b2dd-d65050a9d446 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:50.127098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1348/2704] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:50 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:50.131637 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e867fb-4a1f-45cd-aa93-0454c25842bd admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:50.143121 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e867fb-4a1f-45cd-aa93-0454c25842bd admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:50.148132 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb7cde0-40a8-4944-8484-18920b08507b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:50.148828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb7cde0-40a8-4944-8484-18920b08507b admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:50.149103 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb7cde0-40a8-4944-8484-18920b08507b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:50.149373 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb7cde0-40a8-4944-8484-18920b08507b admin admin] PATH_INFO: `/v3/projects/e9ef1ade541b45358f789ea1368d4599/users/e0080e26588f483fa337ee97b62c1275/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:50.167998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb7cde0-40a8-4944-8484-18920b08507b admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9ef1ade541b45358f789ea1368d4599, user_id=e0080e26588f483fa337ee97b62c1275, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:50.170788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb7cde0-40a8-4944-8484-18920b08507b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:50.191807 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1357/2705] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:18:50 2022] PUT /identity/v3/projects/e9ef1ade541b45358f789ea1368d4599/users/e0080e26588f483fa337ee97b62c1275/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:50.205045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b7c105-e260-4624-9371-5bc1228ea7b4 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:50.205326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b7c105-e260-4624-9371-5bc1228ea7b4 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:50.205585 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b7c105-e260-4624-9371-5bc1228ea7b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:50.244799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-07b7c105-e260-4624-9371-5bc1228ea7b4 None None] MFA Rules not processed for user `e0080e26588f483fa337ee97b62c1275`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:50.246123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-07b7c105-e260-4624-9371-5bc1228ea7b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:50.357032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1349/2706] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:50 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:50.376376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb7cde0-40a8-4944-8484-18920b08507b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:50.379064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8cb7cde0-40a8-4944-8484-18920b08507b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:50.418472 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:INSERT=4,keystone:DELETE=8,keystone:SELECT=88 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:50.472496 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb7cde0-40a8-4944-8484-18920b08507b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:50.476596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3985cbdc-93c0-4ae1-9717-ed91dab56573 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:50.477278 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3985cbdc-93c0-4ae1-9717-ed91dab56573 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:50.477503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3985cbdc-93c0-4ae1-9717-ed91dab56573 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:50.477776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3985cbdc-93c0-4ae1-9717-ed91dab56573 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:50.490635 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3985cbdc-93c0-4ae1-9717-ed91dab56573 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:50.493066 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3985cbdc-93c0-4ae1-9717-ed91dab56573 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:50.507862 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1358/2707] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:50 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:18:50.538347 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b7c105-e260-4624-9371-5bc1228ea7b4 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:50.657393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b7c105-e260-4624-9371-5bc1228ea7b4 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:50.665145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:50.666253 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:50.666734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:50.667263 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] PATH_INFO: `/v3/users/e0080e26588f483fa337ee97b62c1275` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:50.669546 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=e0080e26588f483fa337ee97b62c1275)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:50.673512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:50.685118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:50.753433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e0080e26588f483fa337ee97b62c1275'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:50.754232 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e0080e26588f483fa337ee97b62c1275'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:50.764297 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e0080e26588f483fa337ee97b62c1275'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:50.785324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e0080e26588f483fa337ee97b62c1275'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:50.796806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e0080e26588f483fa337ee97b62c1275'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:50.807640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1350/2708] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:18:50 2022] DELETE /identity/v3/users/e0080e26588f483fa337ee97b62c1275 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:50.814719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5f28ad-e41a-4dfb-8fd6-dd721708d0a0 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:50.815295 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5f28ad-e41a-4dfb-8fd6-dd721708d0a0 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:50.815598 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5f28ad-e41a-4dfb-8fd6-dd721708d0a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:50.840225 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-1f5f28ad-e41a-4dfb-8fd6-dd721708d0a0 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:50.841491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1f5f28ad-e41a-4dfb-8fd6-dd721708d0a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:50.912595 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1359/2709] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:50.923325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:50.925130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:50.999155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7f75c0-0a0f-4d19-9875-1df583ba48cc admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:51.003288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c245df-3c6d-4b24-871b-840e2d25a94e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:51.003958 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c245df-3c6d-4b24-871b-840e2d25a94e service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:51.004233 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c245df-3c6d-4b24-871b-840e2d25a94e service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:51.004499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c245df-3c6d-4b24-871b-840e2d25a94e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:51.015402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c245df-3c6d-4b24-871b-840e2d25a94e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:51.016947 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c245df-3c6d-4b24-871b-840e2d25a94e service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:51.028144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1351/2710] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:18:51.475938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5f28ad-e41a-4dfb-8fd6-dd721708d0a0 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:51.478389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-1f5f28ad-e41a-4dfb-8fd6-dd721708d0a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:51.493085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5f28ad-e41a-4dfb-8fd6-dd721708d0a0 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:51.502661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19de69a-4bd6-47fc-bc66-cf60786d56ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:51.503605 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19de69a-4bd6-47fc-bc66-cf60786d56ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:51.504017 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19de69a-4bd6-47fc-bc66-cf60786d56ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:51.504429 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19de69a-4bd6-47fc-bc66-cf60786d56ee admin admin] PATH_INFO: `/v3/projects/e9ef1ade541b45358f789ea1368d4599` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:51.506820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19de69a-4bd6-47fc-bc66-cf60786d56ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9ef1ade541b45358f789ea1368d4599)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:51.509405 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19de69a-4bd6-47fc-bc66-cf60786d56ee admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:51.598516 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d19de69a-4bd6-47fc-bc66-cf60786d56ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9ef1ade541b45358f789ea1368d4599', 'request_id': 'req-d19de69a-4bd6-47fc-bc66-cf60786d56ee'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:51.611202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d19de69a-4bd6-47fc-bc66-cf60786d56ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9ef1ade541b45358f789ea1368d4599', 'request_id': 'req-d19de69a-4bd6-47fc-bc66-cf60786d56ee'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:51.623824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d19de69a-4bd6-47fc-bc66-cf60786d56ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9ef1ade541b45358f789ea1368d4599', 'request_id': 'req-d19de69a-4bd6-47fc-bc66-cf60786d56ee'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:51.646673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d19de69a-4bd6-47fc-bc66-cf60786d56ee admin admin] The token cache is being invalidate because project e9ef1ade541b45358f789ea1368d4599 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:18:51.647195 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d19de69a-4bd6-47fc-bc66-cf60786d56ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:51.648447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-d19de69a-4bd6-47fc-bc66-cf60786d56ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:51.651147 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1360/2711] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:18:51 2022] DELETE /identity/v3/projects/e9ef1ade541b45358f789ea1368d4599 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:18:51.657478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b3c85-3266-419e-8514-1a89e6107b95 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:51.658066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b3c85-3266-419e-8514-1a89e6107b95 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:51.658066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48b3c85-3266-419e-8514-1a89e6107b95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:51.683975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-f48b3c85-3266-419e-8514-1a89e6107b95 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:51.685334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-f48b3c85-3266-419e-8514-1a89e6107b95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:51.767666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1352/2712] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:51.780277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d1adf5-86a4-46b9-85f9-af7d4c7076fb None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:51.780647 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d1adf5-86a4-46b9-85f9-af7d4c7076fb None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:51.781008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d1adf5-86a4-46b9-85f9-af7d4c7076fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:51.803091 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-13d1adf5-86a4-46b9-85f9-af7d4c7076fb None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:51.804382 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-13d1adf5-86a4-46b9-85f9-af7d4c7076fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:51.816089 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1361/2713] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:18:51.820673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48b3c85-3266-419e-8514-1a89e6107b95 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:51.834637 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48b3c85-3266-419e-8514-1a89e6107b95 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:51.840060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9476d39-62aa-4574-adde-59d3a4b49e0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:51.840805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9476d39-62aa-4574-adde-59d3a4b49e0a admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:51.841083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9476d39-62aa-4574-adde-59d3a4b49e0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:51.841356 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9476d39-62aa-4574-adde-59d3a4b49e0a admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:51.842084 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9476d39-62aa-4574-adde-59d3a4b49e0a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:18:51.842397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9476d39-62aa-4574-adde-59d3a4b49e0a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:51.844096 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9476d39-62aa-4574-adde-59d3a4b49e0a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:51.864791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1353/2714] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:18:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:18:51.871090 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d1adf5-86a4-46b9-85f9-af7d4c7076fb None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:51.883288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d1adf5-86a4-46b9-85f9-af7d4c7076fb None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:51.889515 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962e74f7-914e-4501-83b6-957b9f307a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:51.890519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962e74f7-914e-4501-83b6-957b9f307a03 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:51.890924 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962e74f7-914e-4501-83b6-957b9f307a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:51.891199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962e74f7-914e-4501-83b6-957b9f307a03 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:51.892429 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962e74f7-914e-4501-83b6-957b9f307a03 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:51.893878 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962e74f7-914e-4501-83b6-957b9f307a03 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:51.937379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1362/2715] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:18:51 2022] POST /identity/v3/projects => generated 373 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:18:51.942363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9476d39-62aa-4574-adde-59d3a4b49e0a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:51.957651 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9476d39-62aa-4574-adde-59d3a4b49e0a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:51.984808 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:52.061499 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3935bfe-30c7-4e85-aefa-b9917a7a652d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:52.062063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3935bfe-30c7-4e85-aefa-b9917a7a652d admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:52.062344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3935bfe-30c7-4e85-aefa-b9917a7a652d admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:52.062618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3935bfe-30c7-4e85-aefa-b9917a7a652d admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:52.063326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3935bfe-30c7-4e85-aefa-b9917a7a652d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:52.066172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3935bfe-30c7-4e85-aefa-b9917a7a652d admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:52.099889 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1354/2716] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:18:51 2022] POST /identity/v3/users => generated 287 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:18:52.104957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962e74f7-914e-4501-83b6-957b9f307a03 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:52.118948 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962e74f7-914e-4501-83b6-957b9f307a03 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:52.126197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9ceef2-0281-407c-9eb2-11317e828b5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:52.126936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9ceef2-0281-407c-9eb2-11317e828b5b admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:52.127221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9ceef2-0281-407c-9eb2-11317e828b5b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:52.127531 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9ceef2-0281-407c-9eb2-11317e828b5b admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:52.128272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9ceef2-0281-407c-9eb2-11317e828b5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:52.129931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9ceef2-0281-407c-9eb2-11317e828b5b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:52.141910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1363/2717] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:52 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:18:52.146907 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3935bfe-30c7-4e85-aefa-b9917a7a652d admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:52.158371 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3935bfe-30c7-4e85-aefa-b9917a7a652d admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:52.173193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20886f00-88ed-40d4-9046-8cd8f82cd3f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:52.173450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20886f00-88ed-40d4-9046-8cd8f82cd3f2 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:52.173761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20886f00-88ed-40d4-9046-8cd8f82cd3f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:52.174240 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20886f00-88ed-40d4-9046-8cd8f82cd3f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:52.175640 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20886f00-88ed-40d4-9046-8cd8f82cd3f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:52.178984 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20886f00-88ed-40d4-9046-8cd8f82cd3f2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:52.196708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1355/2718] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:18:52 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:18:52.203134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9ceef2-0281-407c-9eb2-11317e828b5b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:52.210085 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:52.214629 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9ceef2-0281-407c-9eb2-11317e828b5b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:52.223708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd14e4a-39a5-43f0-8cdc-209cb0a7827b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:52.224904 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd14e4a-39a5-43f0-8cdc-209cb0a7827b admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:52.225483 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd14e4a-39a5-43f0-8cdc-209cb0a7827b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:52.225886 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd14e4a-39a5-43f0-8cdc-209cb0a7827b admin admin] PATH_INFO: `/v3/projects/d1f102846fe94bba89d2453b2c8598c2/users/8ec068c20e9b4876ac6cd1d36b24f0fd/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:52.247873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd14e4a-39a5-43f0-8cdc-209cb0a7827b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1f102846fe94bba89d2453b2c8598c2, user_id=8ec068c20e9b4876ac6cd1d36b24f0fd, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:52.250627 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd14e4a-39a5-43f0-8cdc-209cb0a7827b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:52.266361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1364/2719] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:18:52 2022] PUT /identity/v3/projects/d1f102846fe94bba89d2453b2c8598c2/users/8ec068c20e9b4876ac6cd1d36b24f0fd/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:52.277605 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f10a262-ec8f-444a-b95c-52918682f8d6 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:52.278034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f10a262-ec8f-444a-b95c-52918682f8d6 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:52.278434 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f10a262-ec8f-444a-b95c-52918682f8d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:52.312771 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-9f10a262-ec8f-444a-b95c-52918682f8d6 None None] MFA Rules not processed for user `8ec068c20e9b4876ac6cd1d36b24f0fd`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:52.314080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9f10a262-ec8f-444a-b95c-52918682f8d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:52.446038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1356/2720] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:52 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:52.462010 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd14e4a-39a5-43f0-8cdc-209cb0a7827b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:52.464656 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-2fd14e4a-39a5-43f0-8cdc-209cb0a7827b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:52.562993 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd14e4a-39a5-43f0-8cdc-209cb0a7827b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:52.568881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70128e1f-e82e-4f0c-9b7c-c789196691b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:52.568881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70128e1f-e82e-4f0c-9b7c-c789196691b0 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:52.569308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70128e1f-e82e-4f0c-9b7c-c789196691b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:52.569308 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70128e1f-e82e-4f0c-9b7c-c789196691b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:52.583021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70128e1f-e82e-4f0c-9b7c-c789196691b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:52.585970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70128e1f-e82e-4f0c-9b7c-c789196691b0 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:52.603194 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1365/2721] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:52 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:18:52.632608 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f10a262-ec8f-444a-b95c-52918682f8d6 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:52.710130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f10a262-ec8f-444a-b95c-52918682f8d6 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:52.716207 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:52.716902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:52.717324 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:52.717603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] PATH_INFO: `/v3/users/8ec068c20e9b4876ac6cd1d36b24f0fd` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:52.718874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ec068c20e9b4876ac6cd1d36b24f0fd)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:52.721004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:52.756098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:52.781624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ec068c20e9b4876ac6cd1d36b24f0fd'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:52.782325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ec068c20e9b4876ac6cd1d36b24f0fd'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:52.791328 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ec068c20e9b4876ac6cd1d36b24f0fd'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:52.811603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ec068c20e9b4876ac6cd1d36b24f0fd'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:52.824687 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ec068c20e9b4876ac6cd1d36b24f0fd'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:52.834685 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1357/2722] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:18:52 2022] DELETE /identity/v3/users/8ec068c20e9b4876ac6cd1d36b24f0fd => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:52.840826 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bdccca-863d-4652-8d29-c71a30670044 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:52.841775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bdccca-863d-4652-8d29-c71a30670044 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:52.841775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bdccca-863d-4652-8d29-c71a30670044 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:52.869317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-34bdccca-863d-4652-8d29-c71a30670044 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:18:52.870567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-34bdccca-863d-4652-8d29-c71a30670044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:52.939445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1366/2723] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:18:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:18:52.950996 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:52.952755 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:53.023153 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c210871-2737-4d64-a72a-cd1926ae5867 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:53.028799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a1321e-b868-4fa3-8cb8-5b31eae4cad9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:53.029447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a1321e-b868-4fa3-8cb8-5b31eae4cad9 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:53.029725 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a1321e-b868-4fa3-8cb8-5b31eae4cad9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:53.029994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a1321e-b868-4fa3-8cb8-5b31eae4cad9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:53.040701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a1321e-b868-4fa3-8cb8-5b31eae4cad9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:53.042204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a1321e-b868-4fa3-8cb8-5b31eae4cad9 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:53.054061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1358/2724] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:18:53.395836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bdccca-863d-4652-8d29-c71a30670044 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:53.397640 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-34bdccca-863d-4652-8d29-c71a30670044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:53.419327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bdccca-863d-4652-8d29-c71a30670044 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:53.428311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:53.429295 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:53.429811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:53.430248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] PATH_INFO: `/v3/projects/d1f102846fe94bba89d2453b2c8598c2` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:53.432528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1f102846fe94bba89d2453b2c8598c2)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:53.435651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:53.500307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1f102846fe94bba89d2453b2c8598c2', 'request_id': 'req-515f5b66-68e1-4194-9d51-b76b092d2197'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:53.512247 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1f102846fe94bba89d2453b2c8598c2', 'request_id': 'req-515f5b66-68e1-4194-9d51-b76b092d2197'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:53.521288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1f102846fe94bba89d2453b2c8598c2', 'request_id': 'req-515f5b66-68e1-4194-9d51-b76b092d2197'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:53.538412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] The token cache is being invalidate because project d1f102846fe94bba89d2453b2c8598c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:18:53.538855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:53.539696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:18:53.547345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1367/2725] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:18:53 2022] DELETE /identity/v3/projects/d1f102846fe94bba89d2453b2c8598c2 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:18:55.743763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a1321e-b868-4fa3-8cb8-5b31eae4cad9 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:55.756642 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a1321e-b868-4fa3-8cb8-5b31eae4cad9 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:55.818086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbf8656-b524-4fd7-84b9-61ce07a31183 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:55.818630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbf8656-b524-4fd7-84b9-61ce07a31183 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:55.818898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbf8656-b524-4fd7-84b9-61ce07a31183 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:55.819162 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbf8656-b524-4fd7-84b9-61ce07a31183 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:55.821071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2bbf8656-b524-4fd7-84b9-61ce07a31183 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:55.896076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbf8656-b524-4fd7-84b9-61ce07a31183 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:55.897715 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbf8656-b524-4fd7-84b9-61ce07a31183 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:55.903226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:55.909923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1359/2726] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:55 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:18:57.203061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:18:57.206202 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:57.232124 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:18:57.364875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f5b66-68e1-4194-9d51-b76b092d2197 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:18:57.374258 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b165916c-3095-4aba-96b0-6ab12eef93f5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:18:57.375104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b165916c-3095-4aba-96b0-6ab12eef93f5 service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:18:57.375617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b165916c-3095-4aba-96b0-6ab12eef93f5 service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:18:57.376282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b165916c-3095-4aba-96b0-6ab12eef93f5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:18:57.380702 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b165916c-3095-4aba-96b0-6ab12eef93f5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:18:57.401321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b165916c-3095-4aba-96b0-6ab12eef93f5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:18:57.404289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b165916c-3095-4aba-96b0-6ab12eef93f5 service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:18:57.429538 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1368/2727] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:18:57 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:19:02.125512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbf8656-b524-4fd7-84b9-61ce07a31183 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:02.127845 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2bbf8656-b524-4fd7-84b9-61ce07a31183 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:02.128322 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b165916c-3095-4aba-96b0-6ab12eef93f5 service nova] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:02.129743 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b165916c-3095-4aba-96b0-6ab12eef93f5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:02.146946 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b165916c-3095-4aba-96b0-6ab12eef93f5 service nova] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:02.148961 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbf8656-b524-4fd7-84b9-61ce07a31183 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:02.152568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064b7892-74d2-46a4-b124-83ce604b8a5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:02.153668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064b7892-74d2-46a4-b124-83ce604b8a5a service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:02.154132 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064b7892-74d2-46a4-b124-83ce604b8a5a service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:02.154678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064b7892-74d2-46a4-b124-83ce604b8a5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:02.156005 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f1e686-8571-4d9f-bcae-2e1ef2ae6bc3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:02.156654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f1e686-8571-4d9f-bcae-2e1ef2ae6bc3 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:02.157121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f1e686-8571-4d9f-bcae-2e1ef2ae6bc3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:02.157587 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f1e686-8571-4d9f-bcae-2e1ef2ae6bc3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:02.157991 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-064b7892-74d2-46a4-b124-83ce604b8a5a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:02.161064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b3f1e686-8571-4d9f-bcae-2e1ef2ae6bc3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:02.169894 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064b7892-74d2-46a4-b124-83ce604b8a5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:02.171464 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064b7892-74d2-46a4-b124-83ce604b8a5a service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:02.180004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f1e686-8571-4d9f-bcae-2e1ef2ae6bc3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:02.183641 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f1e686-8571-4d9f-bcae-2e1ef2ae6bc3 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:02.184297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1369/2728] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:19:02 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:19:02.198589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1360/2729] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:19:02 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:19:06.405726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064b7892-74d2-46a4-b124-83ce604b8a5a service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:06.407325 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-064b7892-74d2-46a4-b124-83ce604b8a5a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:06.419701 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064b7892-74d2-46a4-b124-83ce604b8a5a service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:06.423676 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0827b45d-4b87-421b-bbff-abdb6eb3b204 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:06.424365 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0827b45d-4b87-421b-bbff-abdb6eb3b204 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:06.424687 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0827b45d-4b87-421b-bbff-abdb6eb3b204 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:06.425001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0827b45d-4b87-421b-bbff-abdb6eb3b204 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:06.427597 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0827b45d-4b87-421b-bbff-abdb6eb3b204 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:06.504011 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0827b45d-4b87-421b-bbff-abdb6eb3b204 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:06.505895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0827b45d-4b87-421b-bbff-abdb6eb3b204 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:06.518010 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1370/2730] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:19:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:19:11.060553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f1e686-8571-4d9f-bcae-2e1ef2ae6bc3 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:11.080248 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f1e686-8571-4d9f-bcae-2e1ef2ae6bc3 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:11.088537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efe4650-4d52-4de8-a4d8-cedc4c52d302 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:11.089396 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efe4650-4d52-4de8-a4d8-cedc4c52d302 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:11.089768 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efe4650-4d52-4de8-a4d8-cedc4c52d302 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:11.090153 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efe4650-4d52-4de8-a4d8-cedc4c52d302 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:11.094122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0efe4650-4d52-4de8-a4d8-cedc4c52d302 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:11.121337 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0efe4650-4d52-4de8-a4d8-cedc4c52d302 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:11.124700 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0efe4650-4d52-4de8-a4d8-cedc4c52d302 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:11.149144 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1361/2731] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:19:11 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:19:15.568488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0827b45d-4b87-421b-bbff-abdb6eb3b204 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:15.571425 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0827b45d-4b87-421b-bbff-abdb6eb3b204 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:15.595401 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0827b45d-4b87-421b-bbff-abdb6eb3b204 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:15.602061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:15.602779 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:15.603097 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:15.603498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] PATH_INFO: `/v3/users/cb4bfd7ea6664747a6e0693a2fd11e3b` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:15.605373 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb4bfd7ea6664747a6e0693a2fd11e3b)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:15.607612 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:15.675037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb4bfd7ea6664747a6e0693a2fd11e3b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:15.684693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb4bfd7ea6664747a6e0693a2fd11e3b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:15.685465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb4bfd7ea6664747a6e0693a2fd11e3b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:15.695243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:19:15.701592 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb4bfd7ea6664747a6e0693a2fd11e3b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:15.702537 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb4bfd7ea6664747a6e0693a2fd11e3b'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:15.719658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1371/2732] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:19:15 2022] DELETE /identity/v3/users/cb4bfd7ea6664747a6e0693a2fd11e3b => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:19:15.984590 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efe4650-4d52-4de8-a4d8-cedc4c52d302 service nova] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:15.997326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efe4650-4d52-4de8-a4d8-cedc4c52d302 service nova] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:16.070632 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:16.071380 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:16.071721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:16.072045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] PATH_INFO: `/v3/projects/5ca9597665cd4068b0a111136d6e0489` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:16.074011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ca9597665cd4068b0a111136d6e0489)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:16.076191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:16.132875 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ca9597665cd4068b0a111136d6e0489', 'request_id': 'req-250bf6a1-4606-4c1a-a308-2d8e63a00d31'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.141090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ca9597665cd4068b0a111136d6e0489', 'request_id': 'req-250bf6a1-4606-4c1a-a308-2d8e63a00d31'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.152356 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ca9597665cd4068b0a111136d6e0489', 'request_id': 'req-250bf6a1-4606-4c1a-a308-2d8e63a00d31'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.164606 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] The token cache is being invalidate because project 5ca9597665cd4068b0a111136d6e0489 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:19:16.164929 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.165555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.167157 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1362/2733] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:19:15 2022] DELETE /identity/v3/projects/5ca9597665cd4068b0a111136d6e0489 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:19:16.171695 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:16.173403 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:16.187690 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0843cf-c306-49aa-890c-9e91bff53656 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:16.254754 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7dec8d-74b1-4e74-9bda-d28445812796 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:16.255445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7dec8d-74b1-4e74-9bda-d28445812796 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:16.255727 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7dec8d-74b1-4e74-9bda-d28445812796 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:16.256002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7dec8d-74b1-4e74-9bda-d28445812796 admin admin] PATH_INFO: `/v3/users/002e3e2e5b7740c58c65072e11a80e51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:16.257281 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7dec8d-74b1-4e74-9bda-d28445812796 admin admin] RBAC: Authorizing `identity:delete_user(user_id=002e3e2e5b7740c58c65072e11a80e51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:16.259234 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7dec8d-74b1-4e74-9bda-d28445812796 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:16.315541 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bc7dec8d-74b1-4e74-9bda-d28445812796 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '002e3e2e5b7740c58c65072e11a80e51'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.324707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bc7dec8d-74b1-4e74-9bda-d28445812796 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '002e3e2e5b7740c58c65072e11a80e51'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.325570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bc7dec8d-74b1-4e74-9bda-d28445812796 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '002e3e2e5b7740c58c65072e11a80e51'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.341062 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bc7dec8d-74b1-4e74-9bda-d28445812796 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '002e3e2e5b7740c58c65072e11a80e51'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.341856 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bc7dec8d-74b1-4e74-9bda-d28445812796 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '002e3e2e5b7740c58c65072e11a80e51'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.358012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1372/2734] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:19:16 2022] DELETE /identity/v3/users/002e3e2e5b7740c58c65072e11a80e51 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:19:16.587639 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:16.589456 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:16.624718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250bf6a1-4606-4c1a-a308-2d8e63a00d31 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:16.715062 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:16.715925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:16.716254 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:16.716565 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] PATH_INFO: `/v3/projects/1ebe85a1d3e9405fb63f4ce48c29efa6` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:16.718151 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ebe85a1d3e9405fb63f4ce48c29efa6)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:16.720545 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:16.729121 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:19:16.796309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ebe85a1d3e9405fb63f4ce48c29efa6', 'request_id': 'req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.810333 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ebe85a1d3e9405fb63f4ce48c29efa6', 'request_id': 'req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.824694 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ebe85a1d3e9405fb63f4ce48c29efa6', 'request_id': 'req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.839136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] The token cache is being invalidate because project 1ebe85a1d3e9405fb63f4ce48c29efa6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:19:16.839581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.840402 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:16.842343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1363/2735] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:19:16 2022] DELETE /identity/v3/projects/1ebe85a1d3e9405fb63f4ce48c29efa6 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:19:22.276299 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd6d80e-7219-48c6-85c1-cecdcfcbe972 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:22.277507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd6d80e-7219-48c6-85c1-cecdcfcbe972 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:22.277507 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd6d80e-7219-48c6-85c1-cecdcfcbe972 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:22.298473 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-dcd6d80e-7219-48c6-85c1-cecdcfcbe972 None None] MFA Rules not processed for user `943af9a17a554646808ac2f2598a9adf`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:19:22.299772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-dcd6d80e-7219-48c6-85c1-cecdcfcbe972 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:22.364863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1373/2736] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:19:22 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:19:22.653122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:22.654658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:22.725562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621baf4f-f5cc-429f-a0f3-79d6c2ee86fc admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:22.729918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0552dd7b-090f-4f13-ad95-cd4f9225a27e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:22.730623 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0552dd7b-090f-4f13-ad95-cd4f9225a27e service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:22.730992 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0552dd7b-090f-4f13-ad95-cd4f9225a27e service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:22.731312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0552dd7b-090f-4f13-ad95-cd4f9225a27e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:22.733449 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0552dd7b-090f-4f13-ad95-cd4f9225a27e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:22.745822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0552dd7b-090f-4f13-ad95-cd4f9225a27e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:22.747455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0552dd7b-090f-4f13-ad95-cd4f9225a27e service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:22.756891 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1364/2737] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:19:22 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:19:23.140282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd6d80e-7219-48c6-85c1-cecdcfcbe972 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:23.142197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-dcd6d80e-7219-48c6-85c1-cecdcfcbe972 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:23.159288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:19:23.269392 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd6d80e-7219-48c6-85c1-cecdcfcbe972 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:23.274742 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95058ca8-6116-4dad-affa-a36397f4a946 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:23.275402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95058ca8-6116-4dad-affa-a36397f4a946 service placement] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:23.275682 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95058ca8-6116-4dad-affa-a36397f4a946 service placement] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:23.275949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95058ca8-6116-4dad-affa-a36397f4a946 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:23.279274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-95058ca8-6116-4dad-affa-a36397f4a946 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:23.301516 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95058ca8-6116-4dad-affa-a36397f4a946 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:23.303772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95058ca8-6116-4dad-affa-a36397f4a946 service placement] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:23.322127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1374/2738] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:19:23 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:19:30.669954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0552dd7b-090f-4f13-ad95-cd4f9225a27e service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:30.674720 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-0552dd7b-090f-4f13-ad95-cd4f9225a27e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:30.822317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0552dd7b-090f-4f13-ad95-cd4f9225a27e service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:30.830500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a83cdf4-3444-4455-8751-cca733d52801 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:30.831387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a83cdf4-3444-4455-8751-cca733d52801 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:30.831779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a83cdf4-3444-4455-8751-cca733d52801 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:30.832194 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a83cdf4-3444-4455-8751-cca733d52801 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:30.834555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-1a83cdf4-3444-4455-8751-cca733d52801 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:30.852432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a83cdf4-3444-4455-8751-cca733d52801 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:30.856397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a83cdf4-3444-4455-8751-cca733d52801 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:30.877804 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1365/2739] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:19:30 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:19:33.314037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=28 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:19:37.695014 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95058ca8-6116-4dad-affa-a36397f4a946 service placement] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:37.696569 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-95058ca8-6116-4dad-affa-a36397f4a946 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:37.762362 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95058ca8-6116-4dad-affa-a36397f4a946 service placement] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:37.766069 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e6d321-c6b0-4c35-aaed-4c0e00f1e450 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:37.767012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e6d321-c6b0-4c35-aaed-4c0e00f1e450 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:37.767992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e6d321-c6b0-4c35-aaed-4c0e00f1e450 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:37.767992 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e6d321-c6b0-4c35-aaed-4c0e00f1e450 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:37.770954 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b4e6d321-c6b0-4c35-aaed-4c0e00f1e450 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:37.882177 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e6d321-c6b0-4c35-aaed-4c0e00f1e450 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:37.883865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e6d321-c6b0-4c35-aaed-4c0e00f1e450 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:37.907414 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1375/2740] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:19:37 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:19:40.868827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=82,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:19:47.904499 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=61 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:19:53.868607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a83cdf4-3444-4455-8751-cca733d52801 service nova] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:53.879833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a83cdf4-3444-4455-8751-cca733d52801 service nova] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:53.884066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8398bb33-796b-4705-b8e4-80bf95978584 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:53.884742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8398bb33-796b-4705-b8e4-80bf95978584 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:53.885029 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8398bb33-796b-4705-b8e4-80bf95978584 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:53.885303 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8398bb33-796b-4705-b8e4-80bf95978584 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:53.888002 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8398bb33-796b-4705-b8e4-80bf95978584 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:53.970411 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8398bb33-796b-4705-b8e4-80bf95978584 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:53.971920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8398bb33-796b-4705-b8e4-80bf95978584 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:53.984424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1366/2741] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:19:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:19:58.496970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e6d321-c6b0-4c35-aaed-4c0e00f1e450 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:58.497745 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b4e6d321-c6b0-4c35-aaed-4c0e00f1e450 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:58.523319 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e6d321-c6b0-4c35-aaed-4c0e00f1e450 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:58.532794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff414d33-a020-41e1-a340-8fe5f948b9b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:58.534072 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff414d33-a020-41e1-a340-8fe5f948b9b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:58.534567 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff414d33-a020-41e1-a340-8fe5f948b9b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:58.535056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff414d33-a020-41e1-a340-8fe5f948b9b4 admin admin] PATH_INFO: `/v3/users/56732ebe8ee24f3f9ce8f87746e8b161` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:58.537284 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff414d33-a020-41e1-a340-8fe5f948b9b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=56732ebe8ee24f3f9ce8f87746e8b161)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:58.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff414d33-a020-41e1-a340-8fe5f948b9b4 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:58.633438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ff414d33-a020-41e1-a340-8fe5f948b9b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '56732ebe8ee24f3f9ce8f87746e8b161'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:58.645813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ff414d33-a020-41e1-a340-8fe5f948b9b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '56732ebe8ee24f3f9ce8f87746e8b161'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:58.646749 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ff414d33-a020-41e1-a340-8fe5f948b9b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '56732ebe8ee24f3f9ce8f87746e8b161'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:58.667630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ff414d33-a020-41e1-a340-8fe5f948b9b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '56732ebe8ee24f3f9ce8f87746e8b161'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:58.669577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-ff414d33-a020-41e1-a340-8fe5f948b9b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '56732ebe8ee24f3f9ce8f87746e8b161'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:58.693591 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1376/2742] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:19:58 2022] DELETE /identity/v3/users/56732ebe8ee24f3f9ce8f87746e8b161 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:19:58.913654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8398bb33-796b-4705-b8e4-80bf95978584 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:58.929426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8398bb33-796b-4705-b8e4-80bf95978584 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:59.015125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ea4deb-65de-41eb-8edb-4006140abb85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:59.015830 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ea4deb-65de-41eb-8edb-4006140abb85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:59.016148 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ea4deb-65de-41eb-8edb-4006140abb85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:59.016464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ea4deb-65de-41eb-8edb-4006140abb85 admin admin] PATH_INFO: `/v3/projects/80ea98b340d94181b2193f4a79905988` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:59.017756 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ea4deb-65de-41eb-8edb-4006140abb85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=80ea98b340d94181b2193f4a79905988)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:59.020170 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ea4deb-65de-41eb-8edb-4006140abb85 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:59.076308 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-75ea4deb-65de-41eb-8edb-4006140abb85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80ea98b340d94181b2193f4a79905988', 'request_id': 'req-75ea4deb-65de-41eb-8edb-4006140abb85'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:59.084670 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-75ea4deb-65de-41eb-8edb-4006140abb85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80ea98b340d94181b2193f4a79905988', 'request_id': 'req-75ea4deb-65de-41eb-8edb-4006140abb85'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:59.095923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-75ea4deb-65de-41eb-8edb-4006140abb85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80ea98b340d94181b2193f4a79905988', 'request_id': 'req-75ea4deb-65de-41eb-8edb-4006140abb85'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:59.112939 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-75ea4deb-65de-41eb-8edb-4006140abb85 admin admin] The token cache is being invalidate because project 80ea98b340d94181b2193f4a79905988 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:19:59.113378 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-75ea4deb-65de-41eb-8edb-4006140abb85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:59.114139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-75ea4deb-65de-41eb-8edb-4006140abb85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:19:59.115974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1367/2743] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:19:58 2022] DELETE /identity/v3/projects/80ea98b340d94181b2193f4a79905988 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:19:59.122969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7077e42-4521-4f5d-87fa-2b73901f522e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:59.123499 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7077e42-4521-4f5d-87fa-2b73901f522e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:59.124035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7077e42-4521-4f5d-87fa-2b73901f522e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:59.151269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b7077e42-4521-4f5d-87fa-2b73901f522e None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:19:59.152927 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b7077e42-4521-4f5d-87fa-2b73901f522e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:59.242141 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1377/2744] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:19:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:19:59.252507 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575c7caf-96ff-4796-b403-8f9f3c1b94e0 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:59.252938 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575c7caf-96ff-4796-b403-8f9f3c1b94e0 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:59.253314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575c7caf-96ff-4796-b403-8f9f3c1b94e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:59.282500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-575c7caf-96ff-4796-b403-8f9f3c1b94e0 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:19:59.283686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-575c7caf-96ff-4796-b403-8f9f3c1b94e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:59.293581 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1368/2745] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:19:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:19:59.299037 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7077e42-4521-4f5d-87fa-2b73901f522e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:59.312319 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7077e42-4521-4f5d-87fa-2b73901f522e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:59.317882 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e7f815-ea2f-435d-8ddf-2bb59bd862a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:59.318530 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e7f815-ea2f-435d-8ddf-2bb59bd862a8 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:59.318810 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e7f815-ea2f-435d-8ddf-2bb59bd862a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:59.319083 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e7f815-ea2f-435d-8ddf-2bb59bd862a8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:59.319842 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e7f815-ea2f-435d-8ddf-2bb59bd862a8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:19:59.320148 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e7f815-ea2f-435d-8ddf-2bb59bd862a8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:59.321797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e7f815-ea2f-435d-8ddf-2bb59bd862a8 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:59.337196 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1378/2746] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:19:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:19:59.343097 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575c7caf-96ff-4796-b403-8f9f3c1b94e0 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:59.361379 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575c7caf-96ff-4796-b403-8f9f3c1b94e0 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:59.380325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33ac8e9-85b0-416d-846d-a5bada3259ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:59.381338 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33ac8e9-85b0-416d-846d-a5bada3259ca admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:59.381805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33ac8e9-85b0-416d-846d-a5bada3259ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:59.382276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33ac8e9-85b0-416d-846d-a5bada3259ca admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:59.383536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33ac8e9-85b0-416d-846d-a5bada3259ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:59.387143 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33ac8e9-85b0-416d-846d-a5bada3259ca admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:59.421182 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:19:59.444189 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1369/2747] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:19:59 2022] POST /identity/v3/projects => generated 377 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:19:59.450544 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e7f815-ea2f-435d-8ddf-2bb59bd862a8 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:59.461489 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e7f815-ea2f-435d-8ddf-2bb59bd862a8 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:59.530789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047021ac-9067-439c-b48e-ec91c04c4f8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:59.531935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047021ac-9067-439c-b48e-ec91c04c4f8b admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:59.532318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047021ac-9067-439c-b48e-ec91c04c4f8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:59.532782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047021ac-9067-439c-b48e-ec91c04c4f8b admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:59.533521 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047021ac-9067-439c-b48e-ec91c04c4f8b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:59.535972 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047021ac-9067-439c-b48e-ec91c04c4f8b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:59.545648 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:19:59.563624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1379/2748] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:19:59 2022] POST /identity/v3/users => generated 289 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:19:59.567994 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33ac8e9-85b0-416d-846d-a5bada3259ca admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:59.579788 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33ac8e9-85b0-416d-846d-a5bada3259ca admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:59.585133 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede747d7-cd1d-4e5b-b583-ac702b82a08e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:59.585810 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede747d7-cd1d-4e5b-b583-ac702b82a08e admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:59.586130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede747d7-cd1d-4e5b-b583-ac702b82a08e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:59.586514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede747d7-cd1d-4e5b-b583-ac702b82a08e admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:59.587665 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede747d7-cd1d-4e5b-b583-ac702b82a08e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:59.590334 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede747d7-cd1d-4e5b-b583-ac702b82a08e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:59.604385 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1370/2749] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:19:59 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:19:59.611445 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047021ac-9067-439c-b48e-ec91c04c4f8b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:59.628419 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047021ac-9067-439c-b48e-ec91c04c4f8b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:59.636240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c5de31-0b12-4985-935a-41d7081a62eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:59.637182 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c5de31-0b12-4985-935a-41d7081a62eb admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:59.637678 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c5de31-0b12-4985-935a-41d7081a62eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:59.638073 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c5de31-0b12-4985-935a-41d7081a62eb admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:59.639237 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c5de31-0b12-4985-935a-41d7081a62eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:59.642221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c5de31-0b12-4985-935a-41d7081a62eb admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:59.662728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1380/2750] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:19:59 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:19:59.668437 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede747d7-cd1d-4e5b-b583-ac702b82a08e admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:59.680046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede747d7-cd1d-4e5b-b583-ac702b82a08e admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:19:59.685574 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087df2bd-0c6b-4ea5-b823-55552f619690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:19:59.686255 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087df2bd-0c6b-4ea5-b823-55552f619690 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:59.686536 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087df2bd-0c6b-4ea5-b823-55552f619690 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:59.686809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087df2bd-0c6b-4ea5-b823-55552f619690 admin admin] PATH_INFO: `/v3/projects/17c952edcc5440989be309944a4c0b00/users/64fea53260874eae8c13416c97cfa4c7/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:59.706249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087df2bd-0c6b-4ea5-b823-55552f619690 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17c952edcc5440989be309944a4c0b00, user_id=64fea53260874eae8c13416c97cfa4c7, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:19:59.709217 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087df2bd-0c6b-4ea5-b823-55552f619690 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:19:59.727450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1371/2751] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:19:59 2022] PUT /identity/v3/projects/17c952edcc5440989be309944a4c0b00/users/64fea53260874eae8c13416c97cfa4c7/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:19:59.735031 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f03cc-8326-4228-894d-b2f39396bcf9 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:19:59.735618 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f03cc-8326-4228-894d-b2f39396bcf9 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:19:59.736048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f03cc-8326-4228-894d-b2f39396bcf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:19:59.760875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-945f03cc-8326-4228-894d-b2f39396bcf9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:19:59.762283 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-945f03cc-8326-4228-894d-b2f39396bcf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:59.878812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1381/2752] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:19:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:19:59.888978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087df2bd-0c6b-4ea5-b823-55552f619690 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:19:59.890787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-087df2bd-0c6b-4ea5-b823-55552f619690 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:19:59.997463 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087df2bd-0c6b-4ea5-b823-55552f619690 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:00.004476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cf1fa4-af7c-40ef-8444-44afb030b05b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:00.005561 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cf1fa4-af7c-40ef-8444-44afb030b05b service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:00.006055 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cf1fa4-af7c-40ef-8444-44afb030b05b service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:00.006548 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cf1fa4-af7c-40ef-8444-44afb030b05b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:00.024672 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cf1fa4-af7c-40ef-8444-44afb030b05b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:00.028196 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cf1fa4-af7c-40ef-8444-44afb030b05b service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:00.047309 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1372/2753] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:19:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:20:06.894375 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f45c08-caba-499a-a66a-6c5a6aff47ff None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:06.895231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f45c08-caba-499a-a66a-6c5a6aff47ff None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:06.895231 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f45c08-caba-499a-a66a-6c5a6aff47ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:06.951004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-14f45c08-caba-499a-a66a-6c5a6aff47ff None None] MFA Rules not processed for user `64fea53260874eae8c13416c97cfa4c7`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:20:06.953200 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-14f45c08-caba-499a-a66a-6c5a6aff47ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:20:07.063788 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1382/2754] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:20:06 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:20:07.084000 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cf1fa4-af7c-40ef-8444-44afb030b05b service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:07.086414 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-83cf1fa4-af7c-40ef-8444-44afb030b05b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:20:07.198651 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cf1fa4-af7c-40ef-8444-44afb030b05b service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:07.207689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c390b777-f633-480f-bc34-66ec37254ef7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:07.211405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c390b777-f633-480f-bc34-66ec37254ef7 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:07.211405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c390b777-f633-480f-bc34-66ec37254ef7 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:07.211405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c390b777-f633-480f-bc34-66ec37254ef7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:07.223977 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c390b777-f633-480f-bc34-66ec37254ef7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:07.226471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c390b777-f633-480f-bc34-66ec37254ef7 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:07.242779 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1373/2755] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:20:07 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:20:09.399387 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fc846c-08a9-4192-b2e2-32fe3d31a2e2 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:09.400129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fc846c-08a9-4192-b2e2-32fe3d31a2e2 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:09.400129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fc846c-08a9-4192-b2e2-32fe3d31a2e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:09.430348 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-45fc846c-08a9-4192-b2e2-32fe3d31a2e2 None None] MFA Rules not processed for user `64fea53260874eae8c13416c97cfa4c7`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:20:09.431949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-45fc846c-08a9-4192-b2e2-32fe3d31a2e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:20:09.444119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1383/2756] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:20:09 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:20:17.234737 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=77,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:20:19.419737 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:INSERT=3,keystone:SELECT=69 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:20:29.986993 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c390b777-f633-480f-bc34-66ec37254ef7 service nova] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:29.991701 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c390b777-f633-480f-bc34-66ec37254ef7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:20:30.029429 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c390b777-f633-480f-bc34-66ec37254ef7 service nova] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:30.038680 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0177080-4031-4e52-a5de-5a5fe4eb39dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:30.040534 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0177080-4031-4e52-a5de-5a5fe4eb39dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:30.041936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0177080-4031-4e52-a5de-5a5fe4eb39dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:30.041936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0177080-4031-4e52-a5de-5a5fe4eb39dd admin admin] PATH_INFO: `/v3/users/943af9a17a554646808ac2f2598a9adf` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:30.045327 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0177080-4031-4e52-a5de-5a5fe4eb39dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=943af9a17a554646808ac2f2598a9adf)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:30.049707 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0177080-4031-4e52-a5de-5a5fe4eb39dd admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:30.159004 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e0177080-4031-4e52-a5de-5a5fe4eb39dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '943af9a17a554646808ac2f2598a9adf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:30.160313 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e0177080-4031-4e52-a5de-5a5fe4eb39dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '943af9a17a554646808ac2f2598a9adf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:30.176655 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e0177080-4031-4e52-a5de-5a5fe4eb39dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '943af9a17a554646808ac2f2598a9adf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:30.204301 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e0177080-4031-4e52-a5de-5a5fe4eb39dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '943af9a17a554646808ac2f2598a9adf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:30.213975 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-e0177080-4031-4e52-a5de-5a5fe4eb39dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '943af9a17a554646808ac2f2598a9adf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:30.221978 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1374/2757] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:20:29 2022] DELETE /identity/v3/users/943af9a17a554646808ac2f2598a9adf => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:20:30.454951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fc846c-08a9-4192-b2e2-32fe3d31a2e2 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:30.475794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fc846c-08a9-4192-b2e2-32fe3d31a2e2 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:30.547435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30189f41-d630-49d6-9bc2-de88d0489d8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:30.548468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30189f41-d630-49d6-9bc2-de88d0489d8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:30.548935 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30189f41-d630-49d6-9bc2-de88d0489d8d admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:30.549375 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30189f41-d630-49d6-9bc2-de88d0489d8d admin admin] PATH_INFO: `/v3/projects/0f2417d70b00404f8e37d52f694f9a41` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:30.552471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30189f41-d630-49d6-9bc2-de88d0489d8d admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f2417d70b00404f8e37d52f694f9a41)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:30.554708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30189f41-d630-49d6-9bc2-de88d0489d8d admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:30.631900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-30189f41-d630-49d6-9bc2-de88d0489d8d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f2417d70b00404f8e37d52f694f9a41', 'request_id': 'req-30189f41-d630-49d6-9bc2-de88d0489d8d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:30.639950 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-30189f41-d630-49d6-9bc2-de88d0489d8d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f2417d70b00404f8e37d52f694f9a41', 'request_id': 'req-30189f41-d630-49d6-9bc2-de88d0489d8d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:30.649910 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-30189f41-d630-49d6-9bc2-de88d0489d8d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f2417d70b00404f8e37d52f694f9a41', 'request_id': 'req-30189f41-d630-49d6-9bc2-de88d0489d8d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:30.669413 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-30189f41-d630-49d6-9bc2-de88d0489d8d admin admin] The token cache is being invalidate because project 0f2417d70b00404f8e37d52f694f9a41 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:20:30.669738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-30189f41-d630-49d6-9bc2-de88d0489d8d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:30.670408 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-30189f41-d630-49d6-9bc2-de88d0489d8d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:30.673409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1384/2758] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:20:30 2022] DELETE /identity/v3/projects/0f2417d70b00404f8e37d52f694f9a41 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:20:30.677498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51323d0-5f06-4da1-86a4-0463d37d6ebd None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:30.677792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51323d0-5f06-4da1-86a4-0463d37d6ebd None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:30.678066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51323d0-5f06-4da1-86a4-0463d37d6ebd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:30.699809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a51323d0-5f06-4da1-86a4-0463d37d6ebd None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:20:30.701058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a51323d0-5f06-4da1-86a4-0463d37d6ebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:20:30.783259 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1375/2759] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:20:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:20:30.793192 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3ebec3-ffc7-4451-abfa-a6bcf851794d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:30.793468 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3ebec3-ffc7-4451-abfa-a6bcf851794d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:30.793736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3ebec3-ffc7-4451-abfa-a6bcf851794d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:30.817013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-bd3ebec3-ffc7-4451-abfa-a6bcf851794d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:20:30.818317 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-bd3ebec3-ffc7-4451-abfa-a6bcf851794d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:20:30.827251 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1385/2760] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:20:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:20:30.831734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51323d0-5f06-4da1-86a4-0463d37d6ebd None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:30.843466 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51323d0-5f06-4da1-86a4-0463d37d6ebd None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:30.849410 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2f1e9f-c2e0-4f69-ae87-fba83818d5bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:30.850078 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2f1e9f-c2e0-4f69-ae87-fba83818d5bd admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:30.850529 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2f1e9f-c2e0-4f69-ae87-fba83818d5bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:30.850753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2f1e9f-c2e0-4f69-ae87-fba83818d5bd admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:30.851422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2f1e9f-c2e0-4f69-ae87-fba83818d5bd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:20:30.851796 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2f1e9f-c2e0-4f69-ae87-fba83818d5bd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:30.853458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2f1e9f-c2e0-4f69-ae87-fba83818d5bd admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:30.867556 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1376/2761] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:20:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:20:30.872855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3ebec3-ffc7-4451-abfa-a6bcf851794d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:30.887481 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3ebec3-ffc7-4451-abfa-a6bcf851794d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:30.893969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7421ab34-69cf-4f16-bcff-a6f2bd4ce172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:30.894621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7421ab34-69cf-4f16-bcff-a6f2bd4ce172 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:30.894926 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7421ab34-69cf-4f16-bcff-a6f2bd4ce172 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:30.895204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7421ab34-69cf-4f16-bcff-a6f2bd4ce172 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:30.895957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7421ab34-69cf-4f16-bcff-a6f2bd4ce172 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:30.897964 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7421ab34-69cf-4f16-bcff-a6f2bd4ce172 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:30.934863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1386/2762] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:20:30 2022] POST /identity/v3/projects => generated 399 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:20:30.941344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2f1e9f-c2e0-4f69-ae87-fba83818d5bd admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:30.955061 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2f1e9f-c2e0-4f69-ae87-fba83818d5bd admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:31.028654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b10a206-9b18-4f4b-a1a5-aebbb42098b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:31.029852 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b10a206-9b18-4f4b-a1a5-aebbb42098b9 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:31.030315 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b10a206-9b18-4f4b-a1a5-aebbb42098b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:31.030823 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b10a206-9b18-4f4b-a1a5-aebbb42098b9 admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:31.031772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b10a206-9b18-4f4b-a1a5-aebbb42098b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:31.034252 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b10a206-9b18-4f4b-a1a5-aebbb42098b9 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:31.043510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:20:31.062880 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1377/2763] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:20:30 2022] POST /identity/v3/users => generated 300 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:20:31.068422 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7421ab34-69cf-4f16-bcff-a6f2bd4ce172 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:31.082711 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7421ab34-69cf-4f16-bcff-a6f2bd4ce172 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:31.088343 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6cb881-c2ff-4c0a-bcd9-2e73e7e21b61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:31.089134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6cb881-c2ff-4c0a-bcd9-2e73e7e21b61 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:31.089919 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6cb881-c2ff-4c0a-bcd9-2e73e7e21b61 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:31.090206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6cb881-c2ff-4c0a-bcd9-2e73e7e21b61 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:31.090899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6cb881-c2ff-4c0a-bcd9-2e73e7e21b61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:31.092772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6cb881-c2ff-4c0a-bcd9-2e73e7e21b61 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:31.105222 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1387/2764] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:20:31 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:20:31.109559 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b10a206-9b18-4f4b-a1a5-aebbb42098b9 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:31.122752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b10a206-9b18-4f4b-a1a5-aebbb42098b9 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:31.128063 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9da0217-eb8d-49cf-87e2-08918ac90a6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:31.128562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9da0217-eb8d-49cf-87e2-08918ac90a6b admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:31.128877 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9da0217-eb8d-49cf-87e2-08918ac90a6b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:31.129154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9da0217-eb8d-49cf-87e2-08918ac90a6b admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:31.129969 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9da0217-eb8d-49cf-87e2-08918ac90a6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:31.133048 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9da0217-eb8d-49cf-87e2-08918ac90a6b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:31.151173 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1378/2765] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:20:31 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:20:31.155968 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6cb881-c2ff-4c0a-bcd9-2e73e7e21b61 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:31.167312 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6cb881-c2ff-4c0a-bcd9-2e73e7e21b61 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:31.172512 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b7ae42-a12f-47f5-903e-0d4398b3f853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:31.173330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b7ae42-a12f-47f5-903e-0d4398b3f853 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:31.173621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b7ae42-a12f-47f5-903e-0d4398b3f853 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:31.173901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b7ae42-a12f-47f5-903e-0d4398b3f853 admin admin] PATH_INFO: `/v3/projects/7aa26afc8ec64dba8a424f5d912fdf8d/users/a8d9907b061841eda154859d7e8baf49/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:31.192458 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b7ae42-a12f-47f5-903e-0d4398b3f853 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7aa26afc8ec64dba8a424f5d912fdf8d, user_id=a8d9907b061841eda154859d7e8baf49, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:31.195198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b7ae42-a12f-47f5-903e-0d4398b3f853 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:31.210776 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1388/2766] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:20:31 2022] PUT /identity/v3/projects/7aa26afc8ec64dba8a424f5d912fdf8d/users/a8d9907b061841eda154859d7e8baf49/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:20:31.222987 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75471e69-261d-44f6-b33b-5a5eae2cee64 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:31.223270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75471e69-261d-44f6-b33b-5a5eae2cee64 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:31.223582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75471e69-261d-44f6-b33b-5a5eae2cee64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:31.256870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-75471e69-261d-44f6-b33b-5a5eae2cee64 None None] MFA Rules not processed for user `a8d9907b061841eda154859d7e8baf49`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:20:31.258471 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-75471e69-261d-44f6-b33b-5a5eae2cee64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:20:31.336266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1379/2767] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:20:31 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:20:31.348652 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b7ae42-a12f-47f5-903e-0d4398b3f853 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:31.350490 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-53b7ae42-a12f-47f5-903e-0d4398b3f853 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:20:31.393726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:20:31.425706 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b7ae42-a12f-47f5-903e-0d4398b3f853 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:31.432014 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e48087-435c-4843-b91f-da327473c454 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:31.440389 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e48087-435c-4843-b91f-da327473c454 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:31.440767 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e48087-435c-4843-b91f-da327473c454 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:31.441051 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e48087-435c-4843-b91f-da327473c454 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:31.454288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e48087-435c-4843-b91f-da327473c454 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:31.455981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e48087-435c-4843-b91f-da327473c454 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:31.466828 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1389/2768] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:20:31 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:20:39.741893 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75471e69-261d-44f6-b33b-5a5eae2cee64 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:39.859790 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75471e69-261d-44f6-b33b-5a5eae2cee64 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:39.867742 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:39.868862 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:39.869347 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:39.869833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] PATH_INFO: `/v3/users/a8d9907b061841eda154859d7e8baf49` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:39.872274 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8d9907b061841eda154859d7e8baf49)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:39.874721 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:39.944193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8d9907b061841eda154859d7e8baf49'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:39.944888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8d9907b061841eda154859d7e8baf49'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:39.954421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8d9907b061841eda154859d7e8baf49'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:39.970957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:INSERT=4,keystone:SELECT=86,keystone:DELETE=10 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:20:39.973399 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8d9907b061841eda154859d7e8baf49'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:39.986420 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8d9907b061841eda154859d7e8baf49'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:39.993995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1380/2769] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:20:39 2022] DELETE /identity/v3/users/a8d9907b061841eda154859d7e8baf49 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:20:39.999463 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ae4840-8b71-4eb9-aa72-8863d2c0aac3 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:40.000252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ae4840-8b71-4eb9-aa72-8863d2c0aac3 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:40.000252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ae4840-8b71-4eb9-aa72-8863d2c0aac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:40.029329 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-67ae4840-8b71-4eb9-aa72-8863d2c0aac3 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:20:40.031394 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-67ae4840-8b71-4eb9-aa72-8863d2c0aac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:20:40.113494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1390/2770] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:20:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:20:40.124505 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:40.126172 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:20:40.207281 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb3e4a3-265b-4262-9605-e664f40c9c7f admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:40.211287 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99253023-e8a8-420a-acdd-67fbd7dbc8c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:40.212090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99253023-e8a8-420a-acdd-67fbd7dbc8c1 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:40.212407 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99253023-e8a8-420a-acdd-67fbd7dbc8c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:40.212821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99253023-e8a8-420a-acdd-67fbd7dbc8c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:40.227923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99253023-e8a8-420a-acdd-67fbd7dbc8c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:40.231123 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99253023-e8a8-420a-acdd-67fbd7dbc8c1 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:40.245962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1381/2771] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:20:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:20:40.672371 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ae4840-8b71-4eb9-aa72-8863d2c0aac3 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:20:40.675719 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-67ae4840-8b71-4eb9-aa72-8863d2c0aac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:20:40.695770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ae4840-8b71-4eb9-aa72-8863d2c0aac3 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:20:40.705987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:20:40.707502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:20:40.708027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:20:40.708623 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2 admin admin] PATH_INFO: `/v3/projects/7aa26afc8ec64dba8a424f5d912fdf8d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:20:40.711015 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7aa26afc8ec64dba8a424f5d912fdf8d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:20:40.715292 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:20:40.795798 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7aa26afc8ec64dba8a424f5d912fdf8d', 'request_id': 'req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:40.802999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7aa26afc8ec64dba8a424f5d912fdf8d', 'request_id': 'req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:40.810256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7aa26afc8ec64dba8a424f5d912fdf8d', 'request_id': 'req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:40.826116 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2 admin admin] The token cache is being invalidate because project 7aa26afc8ec64dba8a424f5d912fdf8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:20:40.826421 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:40.827065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-7dabfeb7-898f-439b-ac8a-7f78a9d960a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:20:40.829599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1391/2772] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:20:40 2022] DELETE /identity/v3/projects/7aa26afc8ec64dba8a424f5d912fdf8d => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:20:50.240316 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=32 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:20:50.822053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=68,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:21:25.378886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99253023-e8a8-420a-acdd-67fbd7dbc8c1 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:21:25.380876 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-99253023-e8a8-420a-acdd-67fbd7dbc8c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:21:25.472394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99253023-e8a8-420a-acdd-67fbd7dbc8c1 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:21:25.560753 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfb8afe-4266-4272-87c0-841c84c8b5b1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:21:25.561551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfb8afe-4266-4272-87c0-841c84c8b5b1 service placement] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:21:25.561913 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfb8afe-4266-4272-87c0-841c84c8b5b1 service placement] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:21:25.562267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfb8afe-4266-4272-87c0-841c84c8b5b1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:21:25.564424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-fdfb8afe-4266-4272-87c0-841c84c8b5b1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:21:25.585090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfb8afe-4266-4272-87c0-841c84c8b5b1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:21:25.587882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfb8afe-4266-4272-87c0-841c84c8b5b1 service placement] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:21:25.602422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1382/2773] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:21:25 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:21:26.763227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5415ff-6736-4b63-8b73-405a410735a0 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:21:26.763863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5415ff-6736-4b63-8b73-405a410735a0 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:21:26.763863 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5415ff-6736-4b63-8b73-405a410735a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:21:26.786245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-7d5415ff-6736-4b63-8b73-405a410735a0 None None] MFA Rules not processed for user `6a00c618dd284f159d88ad41c0390c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:21:26.787565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7d5415ff-6736-4b63-8b73-405a410735a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:21:26.862338 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1392/2774] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:21:26 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:21:27.250609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfb8afe-4266-4272-87c0-841c84c8b5b1 service placement] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:21:27.252398 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-fdfb8afe-4266-4272-87c0-841c84c8b5b1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:21:27.323220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfb8afe-4266-4272-87c0-841c84c8b5b1 service placement] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:21:27.327068 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa836aae-106a-405c-a06e-7f5f41488c78 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:21:27.327959 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa836aae-106a-405c-a06e-7f5f41488c78 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:21:27.328431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa836aae-106a-405c-a06e-7f5f41488c78 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:21:27.328841 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa836aae-106a-405c-a06e-7f5f41488c78 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:21:27.331314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-aa836aae-106a-405c-a06e-7f5f41488c78 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:21:27.344256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa836aae-106a-405c-a06e-7f5f41488c78 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:21:27.345786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa836aae-106a-405c-a06e-7f5f41488c78 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:21:27.357118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1383/2775] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:21:27 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:21:36.852479 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=25 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:21:37.351703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=84 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:23:06.259119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5415ff-6736-4b63-8b73-405a410735a0 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:23:06.260884 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7d5415ff-6736-4b63-8b73-405a410735a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:23:06.276469 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5415ff-6736-4b63-8b73-405a410735a0 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:23:06.280376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eed2dd2-6c96-4b57-a5f7-0354b4f8978c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:23:06.281020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eed2dd2-6c96-4b57-a5f7-0354b4f8978c service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:23:06.281293 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eed2dd2-6c96-4b57-a5f7-0354b4f8978c service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:23:06.281560 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eed2dd2-6c96-4b57-a5f7-0354b4f8978c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:23:06.283542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8eed2dd2-6c96-4b57-a5f7-0354b4f8978c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:23:06.380162 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eed2dd2-6c96-4b57-a5f7-0354b4f8978c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:23:06.381671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eed2dd2-6c96-4b57-a5f7-0354b4f8978c service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:23:06.397404 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1393/2776] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:23:06 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:23:07.610098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa836aae-106a-405c-a06e-7f5f41488c78 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:23:07.621090 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa836aae-106a-405c-a06e-7f5f41488c78 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:23:07.626137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8035b4b8-4b8c-4aaf-9a13-f905e69e1927 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:23:07.627091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8035b4b8-4b8c-4aaf-9a13-f905e69e1927 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:23:07.627722 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8035b4b8-4b8c-4aaf-9a13-f905e69e1927 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:23:07.628046 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8035b4b8-4b8c-4aaf-9a13-f905e69e1927 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:23:07.630139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8035b4b8-4b8c-4aaf-9a13-f905e69e1927 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:23:07.646626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8035b4b8-4b8c-4aaf-9a13-f905e69e1927 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:23:07.648486 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8035b4b8-4b8c-4aaf-9a13-f905e69e1927 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:23:07.661348 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1384/2777] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:23:07 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:23:16.389811 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=34 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:23:17.655738 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=6 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:23:47.643583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eed2dd2-6c96-4b57-a5f7-0354b4f8978c service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:23:47.645080 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8eed2dd2-6c96-4b57-a5f7-0354b4f8978c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:23:47.713049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eed2dd2-6c96-4b57-a5f7-0354b4f8978c service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:23:47.717092 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38091a2-09dd-4b44-8368-e882624de26f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:23:47.717835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38091a2-09dd-4b44-8368-e882624de26f service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:23:47.718243 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38091a2-09dd-4b44-8368-e882624de26f service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:23:47.718632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38091a2-09dd-4b44-8368-e882624de26f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:23:47.721240 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e38091a2-09dd-4b44-8368-e882624de26f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:23:47.735417 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38091a2-09dd-4b44-8368-e882624de26f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:23:47.737466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38091a2-09dd-4b44-8368-e882624de26f service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:23:47.748701 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1394/2778] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:23:47 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:23:57.744189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=26 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:23:57.947111 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2b9af4-5c3f-43a7-a1c7-c4a52f100751 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:23:57.947973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2b9af4-5c3f-43a7-a1c7-c4a52f100751 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:23:57.947973 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2b9af4-5c3f-43a7-a1c7-c4a52f100751 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:23:57.969792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-5e2b9af4-5c3f-43a7-a1c7-c4a52f100751 None None] MFA Rules not processed for user `6a00c618dd284f159d88ad41c0390c5e`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:23:57.972798 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5e2b9af4-5c3f-43a7-a1c7-c4a52f100751 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:23:57.986704 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1385/2779] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:23:57 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:24:01.490282 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38091a2-09dd-4b44-8368-e882624de26f service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:24:01.509433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38091a2-09dd-4b44-8368-e882624de26f service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:24:01.516197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e14b4a-e7ae-4cd5-9310-67c5bf13bf72 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:24:01.517288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e14b4a-e7ae-4cd5-9310-67c5bf13bf72 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:24:01.517808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e14b4a-e7ae-4cd5-9310-67c5bf13bf72 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:24:01.518430 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e14b4a-e7ae-4cd5-9310-67c5bf13bf72 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:24:01.525273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d7e14b4a-e7ae-4cd5-9310-67c5bf13bf72 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:24:01.541860 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e14b4a-e7ae-4cd5-9310-67c5bf13bf72 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:24:01.544908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e14b4a-e7ae-4cd5-9310-67c5bf13bf72 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:24:01.563396 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1395/2780] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:24:01 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:24:07.962696 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:24:11.555331 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=6 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:24:18.788225 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2b9af4-5c3f-43a7-a1c7-c4a52f100751 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:24:18.799576 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2b9af4-5c3f-43a7-a1c7-c4a52f100751 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:24:18.803425 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51ca608-1a97-4084-92c7-f8d68ea2a4ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:24:18.804140 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51ca608-1a97-4084-92c7-f8d68ea2a4ee service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:24:18.804568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51ca608-1a97-4084-92c7-f8d68ea2a4ee service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:24:18.805026 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51ca608-1a97-4084-92c7-f8d68ea2a4ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:24:18.807568 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a51ca608-1a97-4084-92c7-f8d68ea2a4ee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:24:18.878267 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51ca608-1a97-4084-92c7-f8d68ea2a4ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:24:18.879826 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51ca608-1a97-4084-92c7-f8d68ea2a4ee service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:24:18.891752 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1386/2781] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:24:18 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:24:20.988105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e14b4a-e7ae-4cd5-9310-67c5bf13bf72 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:24:20.990799 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-d7e14b4a-e7ae-4cd5-9310-67c5bf13bf72 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:24:21.021687 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e14b4a-e7ae-4cd5-9310-67c5bf13bf72 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:24:21.032151 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613a9599-8e69-4b9b-a32d-6f38777c4a4a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:24:21.033236 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613a9599-8e69-4b9b-a32d-6f38777c4a4a service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:24:21.033734 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613a9599-8e69-4b9b-a32d-6f38777c4a4a service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:24:21.034233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613a9599-8e69-4b9b-a32d-6f38777c4a4a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:24:21.037418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-613a9599-8e69-4b9b-a32d-6f38777c4a4a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:24:21.060494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613a9599-8e69-4b9b-a32d-6f38777c4a4a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:24:21.064061 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613a9599-8e69-4b9b-a32d-6f38777c4a4a service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:24:21.080475 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1396/2782] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:24:20 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:24:28.886741 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=30 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:24:29.840593 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51ca608-1a97-4084-92c7-f8d68ea2a4ee service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:24:29.843110 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a51ca608-1a97-4084-92c7-f8d68ea2a4ee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:24:29.877064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51ca608-1a97-4084-92c7-f8d68ea2a4ee service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:24:29.891941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a64434e-0341-4f57-9ed0-e6f0c3159770 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:24:29.893112 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a64434e-0341-4f57-9ed0-e6f0c3159770 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:24:29.893668 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a64434e-0341-4f57-9ed0-e6f0c3159770 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:24:29.894206 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a64434e-0341-4f57-9ed0-e6f0c3159770 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:24:29.899128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5a64434e-0341-4f57-9ed0-e6f0c3159770 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:24:29.921635 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a64434e-0341-4f57-9ed0-e6f0c3159770 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:24:29.924651 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a64434e-0341-4f57-9ed0-e6f0c3159770 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:24:29.955952 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1387/2783] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:24:29 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:24:31.074917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=6 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:24:33.078102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ff26bc-417e-4077-b4bf-978313af3da5 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:24:33.078803 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ff26bc-417e-4077-b4bf-978313af3da5 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:24:33.078803 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ff26bc-417e-4077-b4bf-978313af3da5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:24:33.100172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-38ff26bc-417e-4077-b4bf-978313af3da5 None None] MFA Rules not processed for user `64fea53260874eae8c13416c97cfa4c7`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:24:33.101526 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-38ff26bc-417e-4077-b4bf-978313af3da5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:24:33.176380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1397/2784] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:24:33 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:24:33.568410 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a64434e-0341-4f57-9ed0-e6f0c3159770 service nova] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:24:33.579326 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a64434e-0341-4f57-9ed0-e6f0c3159770 service nova] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:24:33.583138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c57591b-ff45-4965-a881-9deebce0a2eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:24:33.583813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c57591b-ff45-4965-a881-9deebce0a2eb service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:24:33.584091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c57591b-ff45-4965-a881-9deebce0a2eb service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:24:33.584367 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c57591b-ff45-4965-a881-9deebce0a2eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:24:33.586353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-5c57591b-ff45-4965-a881-9deebce0a2eb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:24:33.597405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c57591b-ff45-4965-a881-9deebce0a2eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:24:33.598869 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c57591b-ff45-4965-a881-9deebce0a2eb service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:24:33.608600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1388/2785] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:24:33 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:24:33.984781 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ff26bc-417e-4077-b4bf-978313af3da5 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:24:33.986555 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-38ff26bc-417e-4077-b4bf-978313af3da5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:24:34.012450 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ff26bc-417e-4077-b4bf-978313af3da5 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:24:34.018086 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca9d182-ebb0-4faa-8ac5-d01b27541d3d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:24:34.018736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca9d182-ebb0-4faa-8ac5-d01b27541d3d service placement] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:24:34.019027 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca9d182-ebb0-4faa-8ac5-d01b27541d3d service placement] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:24:34.019290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca9d182-ebb0-4faa-8ac5-d01b27541d3d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:24:34.021414 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-eca9d182-ebb0-4faa-8ac5-d01b27541d3d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:24:34.035441 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca9d182-ebb0-4faa-8ac5-d01b27541d3d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:24:34.037482 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca9d182-ebb0-4faa-8ac5-d01b27541d3d service placement] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:24:34.055107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1398/2786] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:24:33 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:24:43.561067 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c57591b-ff45-4965-a881-9deebce0a2eb service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:24:43.579697 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c57591b-ff45-4965-a881-9deebce0a2eb service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:24:43.585740 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e3549c-f0d5-4414-a078-13c921e18454 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:24:43.586485 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e3549c-f0d5-4414-a078-13c921e18454 service nova] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:24:43.586845 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e3549c-f0d5-4414-a078-13c921e18454 service nova] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:24:43.587186 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e3549c-f0d5-4414-a078-13c921e18454 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:24:43.591038 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-06e3549c-f0d5-4414-a078-13c921e18454 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:24:43.611245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e3549c-f0d5-4414-a078-13c921e18454 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:24:43.614330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e3549c-f0d5-4414-a078-13c921e18454 service nova] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:24:43.633735 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1389/2787] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:24:43 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:24:44.045589 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=31 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:24:53.626287 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=18 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:03.328252 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca9d182-ebb0-4faa-8ac5-d01b27541d3d service placement] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:03.345262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca9d182-ebb0-4faa-8ac5-d01b27541d3d service placement] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:03.354692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc79d3f-08c6-4957-a759-40b6a4d95aee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5b0bd572ae7a4c7f9e07fe0b9708aa7e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:03.355586 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc79d3f-08c6-4957-a759-40b6a4d95aee service nova] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:03.356059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc79d3f-08c6-4957-a759-40b6a4d95aee service nova] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:03.356379 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc79d3f-08c6-4957-a759-40b6a4d95aee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:03.358823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8fc79d3f-08c6-4957-a759-40b6a4d95aee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:03.371060 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc79d3f-08c6-4957-a759-40b6a4d95aee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:03.372869 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc79d3f-08c6-4957-a759-40b6a4d95aee service nova] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:03.383985 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1399/2788] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:26:03 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:26:13.378900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=6 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:27.860583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e3549c-f0d5-4414-a078-13c921e18454 service nova] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:27.894092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e3549c-f0d5-4414-a078-13c921e18454 service nova] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:27.901552 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e67a2ac-2b5f-4e5b-91ab-9971f931355d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:27.902686 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e67a2ac-2b5f-4e5b-91ab-9971f931355d service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:27.903171 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e67a2ac-2b5f-4e5b-91ab-9971f931355d service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:27.903739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e67a2ac-2b5f-4e5b-91ab-9971f931355d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:27.908204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9e67a2ac-2b5f-4e5b-91ab-9971f931355d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:28.040662 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e67a2ac-2b5f-4e5b-91ab-9971f931355d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:28.042145 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e67a2ac-2b5f-4e5b-91ab-9971f931355d service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:28.053834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1390/2789] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:26:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:26:31.536794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc79d3f-08c6-4957-a759-40b6a4d95aee service nova] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:31.664592 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc79d3f-08c6-4957-a759-40b6a4d95aee service nova] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:31.670412 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee681fd8-bb15-45c5-98b7-ef555ab7ecfb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:31.671427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee681fd8-bb15-45c5-98b7-ef555ab7ecfb service placement] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:31.671707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee681fd8-bb15-45c5-98b7-ef555ab7ecfb service placement] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:31.672180 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee681fd8-bb15-45c5-98b7-ef555ab7ecfb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:31.695048 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee681fd8-bb15-45c5-98b7-ef555ab7ecfb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:31.705983 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee681fd8-bb15-45c5-98b7-ef555ab7ecfb service placement] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:31.741501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1400/2790] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:26:31 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:26:32.593827 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e67a2ac-2b5f-4e5b-91ab-9971f931355d service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:32.594502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9e67a2ac-2b5f-4e5b-91ab-9971f931355d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:32.638436 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e67a2ac-2b5f-4e5b-91ab-9971f931355d service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:32.648204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7987fcd-e739-4560-bfd0-159f1209849f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:32.648952 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7987fcd-e739-4560-bfd0-159f1209849f service placement] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:32.649451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7987fcd-e739-4560-bfd0-159f1209849f service placement] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:32.649757 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7987fcd-e739-4560-bfd0-159f1209849f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:32.652166 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b7987fcd-e739-4560-bfd0-159f1209849f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:32.669266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7987fcd-e739-4560-bfd0-159f1209849f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:32.671036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7987fcd-e739-4560-bfd0-159f1209849f service placement] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:32.682736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1391/2791] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:26:32 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:26:37.229561 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee681fd8-bb15-45c5-98b7-ef555ab7ecfb service placement] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:37.250435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee681fd8-bb15-45c5-98b7-ef555ab7ecfb service placement] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:37.261581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab036f6-386a-49de-ab3b-173bcda64546 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:37.263546 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab036f6-386a-49de-ab3b-173bcda64546 service placement] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:37.264208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab036f6-386a-49de-ab3b-173bcda64546 service placement] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:37.265098 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab036f6-386a-49de-ab3b-173bcda64546 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:37.268699 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-eab036f6-386a-49de-ab3b-173bcda64546 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:37.293787 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab036f6-386a-49de-ab3b-173bcda64546 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:37.298549 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab036f6-386a-49de-ab3b-173bcda64546 service placement] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:37.318354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7987fcd-e739-4560-bfd0-159f1209849f service placement] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:37.320506 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b7987fcd-e739-4560-bfd0-159f1209849f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:37.336460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1401/2792] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:26:37 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:26:37.354447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7987fcd-e739-4560-bfd0-159f1209849f service placement] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:37.370964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:37.372619 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:37.373762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:37.375392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] PATH_INFO: `/v3/users/6a00c618dd284f159d88ad41c0390c5e` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:37.377915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a00c618dd284f159d88ad41c0390c5e)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:37.382522 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:37.447175 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a00c618dd284f159d88ad41c0390c5e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:37.447895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a00c618dd284f159d88ad41c0390c5e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:37.457292 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a00c618dd284f159d88ad41c0390c5e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:37.476726 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a00c618dd284f159d88ad41c0390c5e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:37.491065 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a00c618dd284f159d88ad41c0390c5e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:37.502092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1392/2793] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:26:37 2022] DELETE /identity/v3/users/6a00c618dd284f159d88ad41c0390c5e => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:26:37.731794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab036f6-386a-49de-ab3b-173bcda64546 service placement] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:37.752297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab036f6-386a-49de-ab3b-173bcda64546 service placement] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:37.827994 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:37.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:37.828968 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:37.829249 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] PATH_INFO: `/v3/projects/145c3d8e13bd4e2793bc930d8638eac1` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:37.830501 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] RBAC: Authorizing `identity:delete_project(project_id=145c3d8e13bd4e2793bc930d8638eac1)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:37.832503 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:37.893165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '145c3d8e13bd4e2793bc930d8638eac1', 'request_id': 'req-83164b3d-62e2-41b4-8f93-7aa676472b78'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:37.900707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '145c3d8e13bd4e2793bc930d8638eac1', 'request_id': 'req-83164b3d-62e2-41b4-8f93-7aa676472b78'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:37.908680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '145c3d8e13bd4e2793bc930d8638eac1', 'request_id': 'req-83164b3d-62e2-41b4-8f93-7aa676472b78'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:37.926692 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] The token cache is being invalidate because project 145c3d8e13bd4e2793bc930d8638eac1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:26:37.927026 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:37.928540 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:37.930454 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1402/2794] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:26:37 2022] DELETE /identity/v3/projects/145c3d8e13bd4e2793bc930d8638eac1 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:26:37.936588 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:37.937816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:37.947451 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:37.956095 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c5a5c5-de4a-4265-a95d-2f0981ed1676 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:38.044267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:38.046289 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:38.053749 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:38.054614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:38.054963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:38.055390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] PATH_INFO: `/v3/users/6cd5aba99ec8492f9891202300dbaaef` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:38.058036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6cd5aba99ec8492f9891202300dbaaef)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:38.060405 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:38.090817 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:38.130873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6cd5aba99ec8492f9891202300dbaaef'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:38.132035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6cd5aba99ec8492f9891202300dbaaef'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:38.143455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6cd5aba99ec8492f9891202300dbaaef'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:38.166058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6cd5aba99ec8492f9891202300dbaaef'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:38.174734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6cd5aba99ec8492f9891202300dbaaef'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:38.182498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1393/2795] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:26:37 2022] DELETE /identity/v3/users/6cd5aba99ec8492f9891202300dbaaef => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:26:38.190552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83164b3d-62e2-41b4-8f93-7aa676472b78 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:38.269047 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d6d01-2bf0-42bf-9bcf-943a870b4b7f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:38.270440 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d6d01-2bf0-42bf-9bcf-943a870b4b7f service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:38.270688 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d6d01-2bf0-42bf-9bcf-943a870b4b7f service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:38.271007 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d6d01-2bf0-42bf-9bcf-943a870b4b7f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:38.274039 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-857d6d01-2bf0-42bf-9bcf-943a870b4b7f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:38.391583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d6d01-2bf0-42bf-9bcf-943a870b4b7f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:38.393498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d6d01-2bf0-42bf-9bcf-943a870b4b7f service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:38.406102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1403/2796] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:26:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:26:38.549836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:38.553138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:38.570122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f17373d-1d06-4a44-8cf9-c8e5864de513 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:38.580343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:38.581954 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:38.582691 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:38.583284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] PATH_INFO: `/v3/projects/e19a1a4c25ff4cd6866816e0600b576d` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:38.586455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e19a1a4c25ff4cd6866816e0600b576d)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:38.590310 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:38.678943 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e19a1a4c25ff4cd6866816e0600b576d', 'request_id': 'req-2a715fcc-b479-492c-899b-6f63cc67a189'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:38.687728 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e19a1a4c25ff4cd6866816e0600b576d', 'request_id': 'req-2a715fcc-b479-492c-899b-6f63cc67a189'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:38.699510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e19a1a4c25ff4cd6866816e0600b576d', 'request_id': 'req-2a715fcc-b479-492c-899b-6f63cc67a189'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:38.718013 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] The token cache is being invalidate because project e19a1a4c25ff4cd6866816e0600b576d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:26:38.718510 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:38.719478 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:38.722135 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1394/2797] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:26:38 2022] DELETE /identity/v3/projects/e19a1a4c25ff4cd6866816e0600b576d => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:26:42.436299 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d6d01-2bf0-42bf-9bcf-943a870b4b7f service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:42.438610 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-857d6d01-2bf0-42bf-9bcf-943a870b4b7f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:42.526614 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d6d01-2bf0-42bf-9bcf-943a870b4b7f service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:42.595753 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=100 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:42.606711 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a657d2b5-2db9-4fa1-82b6-f8bd6327bdc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:42.607830 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a657d2b5-2db9-4fa1-82b6-f8bd6327bdc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:42.608307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a657d2b5-2db9-4fa1-82b6-f8bd6327bdc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:42.608785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a657d2b5-2db9-4fa1-82b6-f8bd6327bdc3 admin admin] PATH_INFO: `/v3/users/64fea53260874eae8c13416c97cfa4c7` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:42.610820 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a657d2b5-2db9-4fa1-82b6-f8bd6327bdc3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=64fea53260874eae8c13416c97cfa4c7)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:42.614205 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a657d2b5-2db9-4fa1-82b6-f8bd6327bdc3 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:42.696655 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a657d2b5-2db9-4fa1-82b6-f8bd6327bdc3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '64fea53260874eae8c13416c97cfa4c7'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:42.708981 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a657d2b5-2db9-4fa1-82b6-f8bd6327bdc3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '64fea53260874eae8c13416c97cfa4c7'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:42.709912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a657d2b5-2db9-4fa1-82b6-f8bd6327bdc3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '64fea53260874eae8c13416c97cfa4c7'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:42.732106 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a657d2b5-2db9-4fa1-82b6-f8bd6327bdc3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '64fea53260874eae8c13416c97cfa4c7'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:42.733740 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a657d2b5-2db9-4fa1-82b6-f8bd6327bdc3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '64fea53260874eae8c13416c97cfa4c7'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:42.753042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1404/2798] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:26:42 2022] DELETE /identity/v3/users/64fea53260874eae8c13416c97cfa4c7 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:26:42.965791 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:42.968092 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:42.976688 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:43.026472 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a715fcc-b479-492c-899b-6f63cc67a189 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:43.094307 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be28e003-4601-477a-806c-61c22c75cd8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:43.094879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28e003-4601-477a-806c-61c22c75cd8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:43.095154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28e003-4601-477a-806c-61c22c75cd8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:43.095459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28e003-4601-477a-806c-61c22c75cd8a admin admin] PATH_INFO: `/v3/projects/17c952edcc5440989be309944a4c0b00` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:43.096711 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be28e003-4601-477a-806c-61c22c75cd8a admin admin] RBAC: Authorizing `identity:delete_project(project_id=17c952edcc5440989be309944a4c0b00)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:43.098657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be28e003-4601-477a-806c-61c22c75cd8a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:43.155834 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-be28e003-4601-477a-806c-61c22c75cd8a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17c952edcc5440989be309944a4c0b00', 'request_id': 'req-be28e003-4601-477a-806c-61c22c75cd8a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:43.169107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-be28e003-4601-477a-806c-61c22c75cd8a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17c952edcc5440989be309944a4c0b00', 'request_id': 'req-be28e003-4601-477a-806c-61c22c75cd8a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:43.181902 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-be28e003-4601-477a-806c-61c22c75cd8a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17c952edcc5440989be309944a4c0b00', 'request_id': 'req-be28e003-4601-477a-806c-61c22c75cd8a'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:43.195387 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-be28e003-4601-477a-806c-61c22c75cd8a admin admin] The token cache is being invalidate because project 17c952edcc5440989be309944a4c0b00 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:26:43.195729 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-be28e003-4601-477a-806c-61c22c75cd8a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:43.196353 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-be28e003-4601-477a-806c-61c22c75cd8a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:43.197906 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1395/2799] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:26:42 2022] DELETE /identity/v3/projects/17c952edcc5440989be309944a4c0b00 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:26:43.204134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ebae63-f43d-45de-8713-8660b07ec50b None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:43.204813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ebae63-f43d-45de-8713-8660b07ec50b None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:43.205355 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ebae63-f43d-45de-8713-8660b07ec50b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:43.228785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-55ebae63-f43d-45de-8713-8660b07ec50b None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:26:43.230149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-55ebae63-f43d-45de-8713-8660b07ec50b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:43.315064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1405/2800] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:26:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:26:43.324298 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1bc577-bac1-4974-b87e-4e2794f0c90d None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:43.324586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1bc577-bac1-4974-b87e-4e2794f0c90d None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:43.324854 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1bc577-bac1-4974-b87e-4e2794f0c90d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:43.356337 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-4c1bc577-bac1-4974-b87e-4e2794f0c90d None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:26:43.358467 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4c1bc577-bac1-4974-b87e-4e2794f0c90d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:43.369450 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1396/2801] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:26:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:26:43.374129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ebae63-f43d-45de-8713-8660b07ec50b None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:43.385601 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ebae63-f43d-45de-8713-8660b07ec50b None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:43.390491 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69eee010-54bc-4e68-94ab-a388e599fb29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:43.391173 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69eee010-54bc-4e68-94ab-a388e599fb29 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:43.391634 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69eee010-54bc-4e68-94ab-a388e599fb29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:43.392023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69eee010-54bc-4e68-94ab-a388e599fb29 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:43.392778 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69eee010-54bc-4e68-94ab-a388e599fb29 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:26:43.393134 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69eee010-54bc-4e68-94ab-a388e599fb29 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:43.394899 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69eee010-54bc-4e68-94ab-a388e599fb29 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:43.409288 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1406/2802] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:26:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:26:43.414032 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1bc577-bac1-4974-b87e-4e2794f0c90d None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:43.425118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1bc577-bac1-4974-b87e-4e2794f0c90d None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:43.431963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6952caa9-0329-40e1-b200-49f1542dfc53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:43.432780 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6952caa9-0329-40e1-b200-49f1542dfc53 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:43.433184 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6952caa9-0329-40e1-b200-49f1542dfc53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:43.433514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6952caa9-0329-40e1-b200-49f1542dfc53 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:43.434575 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6952caa9-0329-40e1-b200-49f1542dfc53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:43.437390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6952caa9-0329-40e1-b200-49f1542dfc53 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:43.474131 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1397/2803] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:26:43 2022] POST /identity/v3/projects => generated 363 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:26:43.479815 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69eee010-54bc-4e68-94ab-a388e599fb29 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:43.493591 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69eee010-54bc-4e68-94ab-a388e599fb29 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:43.568916 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16003528-3909-4480-9175-7c184496b05c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:43.569551 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16003528-3909-4480-9175-7c184496b05c admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:43.569823 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16003528-3909-4480-9175-7c184496b05c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:43.570093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16003528-3909-4480-9175-7c184496b05c admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:43.570784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16003528-3909-4480-9175-7c184496b05c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:43.572812 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16003528-3909-4480-9175-7c184496b05c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:43.583075 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:43.601313 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1407/2804] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:26:43 2022] POST /identity/v3/users => generated 282 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:26:43.606056 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6952caa9-0329-40e1-b200-49f1542dfc53 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:43.624805 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6952caa9-0329-40e1-b200-49f1542dfc53 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:43.633336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22684fef-92c6-46f3-9549-122d6be99d26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:43.634422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22684fef-92c6-46f3-9549-122d6be99d26 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:43.634928 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22684fef-92c6-46f3-9549-122d6be99d26 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:43.635484 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22684fef-92c6-46f3-9549-122d6be99d26 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:43.636816 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22684fef-92c6-46f3-9549-122d6be99d26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:43.639915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22684fef-92c6-46f3-9549-122d6be99d26 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:43.660214 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1398/2805] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:26:43 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:26:43.665074 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16003528-3909-4480-9175-7c184496b05c admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:43.681012 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16003528-3909-4480-9175-7c184496b05c admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:43.686494 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a2eb98-1694-4a6a-abe5-2d036dc4a601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:43.687269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a2eb98-1694-4a6a-abe5-2d036dc4a601 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:43.687621 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a2eb98-1694-4a6a-abe5-2d036dc4a601 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:43.687931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a2eb98-1694-4a6a-abe5-2d036dc4a601 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:43.688667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a2eb98-1694-4a6a-abe5-2d036dc4a601 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:43.690837 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a2eb98-1694-4a6a-abe5-2d036dc4a601 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:43.704553 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1408/2806] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:26:43 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:26:43.709239 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22684fef-92c6-46f3-9549-122d6be99d26 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:43.720424 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22684fef-92c6-46f3-9549-122d6be99d26 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:43.726071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d31184-fe93-4158-a987-89f456ba0c31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:43.726799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d31184-fe93-4158-a987-89f456ba0c31 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:43.727073 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d31184-fe93-4158-a987-89f456ba0c31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:43.727341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d31184-fe93-4158-a987-89f456ba0c31 admin admin] PATH_INFO: `/v3/projects/b9e4a2ff78db4579a9c06ab7bf663b6b/users/79ad6545a8df4e80ae43073fd2344921/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:43.750041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d31184-fe93-4158-a987-89f456ba0c31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9e4a2ff78db4579a9c06ab7bf663b6b, user_id=79ad6545a8df4e80ae43073fd2344921, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:43.753175 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d31184-fe93-4158-a987-89f456ba0c31 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:43.774678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1399/2807] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:26:43 2022] PUT /identity/v3/projects/b9e4a2ff78db4579a9c06ab7bf663b6b/users/79ad6545a8df4e80ae43073fd2344921/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:26:43.785319 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914f3432-f145-42fc-bc87-577c4df2823e None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:43.785640 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914f3432-f145-42fc-bc87-577c4df2823e None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:43.785861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914f3432-f145-42fc-bc87-577c4df2823e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:43.822004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-914f3432-f145-42fc-bc87-577c4df2823e None None] MFA Rules not processed for user `79ad6545a8df4e80ae43073fd2344921`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:26:43.823245 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-914f3432-f145-42fc-bc87-577c4df2823e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:43.907232 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1409/2808] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:26:43 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:26:43.912663 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d31184-fe93-4158-a987-89f456ba0c31 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:43.933456 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:43.998582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d31184-fe93-4158-a987-89f456ba0c31 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:44.004332 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2903a5e-9ddc-413c-8554-e342fb70bb40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:44.005070 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2903a5e-9ddc-413c-8554-e342fb70bb40 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:44.005513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2903a5e-9ddc-413c-8554-e342fb70bb40 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:44.005840 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2903a5e-9ddc-413c-8554-e342fb70bb40 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:44.006569 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2903a5e-9ddc-413c-8554-e342fb70bb40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:44.008621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2903a5e-9ddc-413c-8554-e342fb70bb40 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:44.050630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1400/2809] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:26:43 2022] POST /identity/v3/projects => generated 363 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:26:44.056619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914f3432-f145-42fc-bc87-577c4df2823e None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:44.069064 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914f3432-f145-42fc-bc87-577c4df2823e None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:44.145998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b0140-2e35-40de-bcaa-68e690fa18f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:44.147158 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b0140-2e35-40de-bcaa-68e690fa18f1 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:44.147477 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b0140-2e35-40de-bcaa-68e690fa18f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:44.147744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b0140-2e35-40de-bcaa-68e690fa18f1 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:44.148447 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914b0140-2e35-40de-bcaa-68e690fa18f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:44.150427 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914b0140-2e35-40de-bcaa-68e690fa18f1 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:44.178460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1410/2810] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:26:44 2022] POST /identity/v3/users => generated 288 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:26:44.182553 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2903a5e-9ddc-413c-8554-e342fb70bb40 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:44.200514 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2903a5e-9ddc-413c-8554-e342fb70bb40 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:44.205732 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bcd146-b1c3-4283-a911-da2ed8255108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:44.206448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bcd146-b1c3-4283-a911-da2ed8255108 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:44.206762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bcd146-b1c3-4283-a911-da2ed8255108 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:44.207091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bcd146-b1c3-4283-a911-da2ed8255108 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:44.207916 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bcd146-b1c3-4283-a911-da2ed8255108 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:44.210397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bcd146-b1c3-4283-a911-da2ed8255108 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:44.224713 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1401/2811] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:26:44 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:26:44.229860 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b0140-2e35-40de-bcaa-68e690fa18f1 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:44.244399 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b0140-2e35-40de-bcaa-68e690fa18f1 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:44.249730 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab64460f-87f7-4023-8e89-a206deda8eef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:44.250715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab64460f-87f7-4023-8e89-a206deda8eef admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:44.251104 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab64460f-87f7-4023-8e89-a206deda8eef admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:44.251480 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab64460f-87f7-4023-8e89-a206deda8eef admin admin] PATH_INFO: `/v3/projects/6f5a32f178a04b4e8c2d8f9670322a97/users/f952c1f32dfd4fc099c78ce5dc0e91c4/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:44.269880 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab64460f-87f7-4023-8e89-a206deda8eef admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f5a32f178a04b4e8c2d8f9670322a97, user_id=f952c1f32dfd4fc099c78ce5dc0e91c4, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:44.273321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab64460f-87f7-4023-8e89-a206deda8eef admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:44.289644 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1411/2812] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:26:44 2022] PUT /identity/v3/projects/6f5a32f178a04b4e8c2d8f9670322a97/users/f952c1f32dfd4fc099c78ce5dc0e91c4/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:26:44.293995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bcd146-b1c3-4283-a911-da2ed8255108 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:44.305146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bcd146-b1c3-4283-a911-da2ed8255108 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:44.382502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ea2757-95de-42ae-9dc2-7a1d10dcea1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:44.383138 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ea2757-95de-42ae-9dc2-7a1d10dcea1b admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:44.383445 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ea2757-95de-42ae-9dc2-7a1d10dcea1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:44.383716 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ea2757-95de-42ae-9dc2-7a1d10dcea1b admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:44.384447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ea2757-95de-42ae-9dc2-7a1d10dcea1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:44.386088 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ea2757-95de-42ae-9dc2-7a1d10dcea1b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:44.401284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1402/2813] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:26:44 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:26:44.405995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab64460f-87f7-4023-8e89-a206deda8eef admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:44.417892 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab64460f-87f7-4023-8e89-a206deda8eef admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:44.423814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1e3857-e9b9-44f5-8f7e-c64536ab7c6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:44.424518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1e3857-e9b9-44f5-8f7e-c64536ab7c6e admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:44.424999 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1e3857-e9b9-44f5-8f7e-c64536ab7c6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:44.425345 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1e3857-e9b9-44f5-8f7e-c64536ab7c6e admin admin] PATH_INFO: `/v3/projects/6f5a32f178a04b4e8c2d8f9670322a97/users/f952c1f32dfd4fc099c78ce5dc0e91c4/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:44.427941 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1e3857-e9b9-44f5-8f7e-c64536ab7c6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f5a32f178a04b4e8c2d8f9670322a97, user_id=f952c1f32dfd4fc099c78ce5dc0e91c4, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:44.430802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1e3857-e9b9-44f5-8f7e-c64536ab7c6e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:44.446005 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1412/2814] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:26:44 2022] PUT /identity/v3/projects/6f5a32f178a04b4e8c2d8f9670322a97/users/f952c1f32dfd4fc099c78ce5dc0e91c4/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:26:44.458739 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49fa744-ca2e-46c9-af96-85db486a9f60 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:44.459199 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49fa744-ca2e-46c9-af96-85db486a9f60 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:44.459708 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49fa744-ca2e-46c9-af96-85db486a9f60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:44.503028 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-a49fa744-ca2e-46c9-af96-85db486a9f60 None None] MFA Rules not processed for user `f952c1f32dfd4fc099c78ce5dc0e91c4`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:26:44.505187 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-a49fa744-ca2e-46c9-af96-85db486a9f60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:44.622421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1403/2815] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:26:44 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:26:44.634883 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1e3857-e9b9-44f5-8f7e-c64536ab7c6e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:44.637108 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-de1e3857-e9b9-44f5-8f7e-c64536ab7c6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:44.671020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:INSERT=7,keystone:SELECT=93 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:44.738176 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1e3857-e9b9-44f5-8f7e-c64536ab7c6e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:44.742465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa58675f-c1ed-4f83-b1d7-479b70f3fa8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:44.743125 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa58675f-c1ed-4f83-b1d7-479b70f3fa8a service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:44.743437 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa58675f-c1ed-4f83-b1d7-479b70f3fa8a service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:44.743717 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa58675f-c1ed-4f83-b1d7-479b70f3fa8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:44.754025 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa58675f-c1ed-4f83-b1d7-479b70f3fa8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:44.755583 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa58675f-c1ed-4f83-b1d7-479b70f3fa8a service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:44.766905 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1413/2816] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:26:44 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:26:44.871184 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49fa744-ca2e-46c9-af96-85db486a9f60 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:44.888208 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49fa744-ca2e-46c9-af96-85db486a9f60 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:44.894435 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bb2fc8-6390-4488-913d-36c9fa04a230 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:44.895539 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bb2fc8-6390-4488-913d-36c9fa04a230 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:44.896579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bb2fc8-6390-4488-913d-36c9fa04a230 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:44.897066 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bb2fc8-6390-4488-913d-36c9fa04a230 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:44.905482 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:44.988152 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bb2fc8-6390-4488-913d-36c9fa04a230 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:44.989736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bb2fc8-6390-4488-913d-36c9fa04a230 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:45.000021 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1404/2817] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:26:44 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:26:54.762067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=25 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:54.995276 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=25 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:55.765118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa58675f-c1ed-4f83-b1d7-479b70f3fa8a service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:55.842755 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa58675f-c1ed-4f83-b1d7-479b70f3fa8a service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:55.847856 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:55.848531 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:55.848809 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:55.849149 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] PATH_INFO: `/v3/users/79ad6545a8df4e80ae43073fd2344921` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:55.851001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] RBAC: Authorizing `identity:delete_user(user_id=79ad6545a8df4e80ae43073fd2344921)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:55.853022 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:55.926016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79ad6545a8df4e80ae43073fd2344921'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:55.933987 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79ad6545a8df4e80ae43073fd2344921'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:55.934895 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79ad6545a8df4e80ae43073fd2344921'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:55.949875 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79ad6545a8df4e80ae43073fd2344921'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:55.950668 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79ad6545a8df4e80ae43073fd2344921'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:55.966112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1414/2818] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:26:55 2022] DELETE /identity/v3/users/79ad6545a8df4e80ae43073fd2344921 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:26:55.971213 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb14ad3-0d3a-41c5-a549-24db268f37d3 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:55.971540 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb14ad3-0d3a-41c5-a549-24db268f37d3 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:55.971821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb14ad3-0d3a-41c5-a549-24db268f37d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:56.002879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-fbb14ad3-0d3a-41c5-a549-24db268f37d3 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:26:56.004060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-fbb14ad3-0d3a-41c5-a549-24db268f37d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:56.078900 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1405/2819] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:26:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:26:56.089879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:56.091528 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:56.182114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2b28e3-b1dc-4fb7-9670-26bf07fdd954 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:56.186563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b22b8e9-40df-4d09-8c1b-b2299e59a842 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:56.187197 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b22b8e9-40df-4d09-8c1b-b2299e59a842 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:56.187504 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b22b8e9-40df-4d09-8c1b-b2299e59a842 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:56.187772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b22b8e9-40df-4d09-8c1b-b2299e59a842 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:56.199484 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b22b8e9-40df-4d09-8c1b-b2299e59a842 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:56.201067 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b22b8e9-40df-4d09-8c1b-b2299e59a842 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:56.212717 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1415/2820] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:26:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:26:56.546201 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb14ad3-0d3a-41c5-a549-24db268f37d3 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:56.548840 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-fbb14ad3-0d3a-41c5-a549-24db268f37d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:56.566120 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb14ad3-0d3a-41c5-a549-24db268f37d3 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:56.571661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:56.572422 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:56.572703 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:56.572981 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] PATH_INFO: `/v3/projects/b9e4a2ff78db4579a9c06ab7bf663b6b` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:56.574246 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b9e4a2ff78db4579a9c06ab7bf663b6b)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:56.576229 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:56.633146 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b9e4a2ff78db4579a9c06ab7bf663b6b', 'request_id': 'req-8262ac06-4b0d-4671-97eb-7ef25bd15966'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:56.640295 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b9e4a2ff78db4579a9c06ab7bf663b6b', 'request_id': 'req-8262ac06-4b0d-4671-97eb-7ef25bd15966'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:56.650137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b9e4a2ff78db4579a9c06ab7bf663b6b', 'request_id': 'req-8262ac06-4b0d-4671-97eb-7ef25bd15966'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:56.663106 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] The token cache is being invalidate because project b9e4a2ff78db4579a9c06ab7bf663b6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:26:56.663562 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:56.664244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:56.665821 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1406/2821] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:26:56 2022] DELETE /identity/v3/projects/b9e4a2ff78db4579a9c06ab7bf663b6b => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:26:56.670053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b22b8e9-40df-4d09-8c1b-b2299e59a842 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:56.675166 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:56.681909 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b22b8e9-40df-4d09-8c1b-b2299e59a842 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:56.744901 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c015d9cd-7301-4944-b92b-210d933f89c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:56.745563 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015d9cd-7301-4944-b92b-210d933f89c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:56.745845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015d9cd-7301-4944-b92b-210d933f89c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:56.746119 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015d9cd-7301-4944-b92b-210d933f89c3 admin admin] PATH_INFO: `/v3/users/f952c1f32dfd4fc099c78ce5dc0e91c4` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:56.747337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c015d9cd-7301-4944-b92b-210d933f89c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f952c1f32dfd4fc099c78ce5dc0e91c4)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:56.749318 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c015d9cd-7301-4944-b92b-210d933f89c3 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:56.800035 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c015d9cd-7301-4944-b92b-210d933f89c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f952c1f32dfd4fc099c78ce5dc0e91c4'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:56.806988 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c015d9cd-7301-4944-b92b-210d933f89c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f952c1f32dfd4fc099c78ce5dc0e91c4'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:56.807693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c015d9cd-7301-4944-b92b-210d933f89c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f952c1f32dfd4fc099c78ce5dc0e91c4'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:56.820487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c015d9cd-7301-4944-b92b-210d933f89c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f952c1f32dfd4fc099c78ce5dc0e91c4'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:56.821422 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-c015d9cd-7301-4944-b92b-210d933f89c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f952c1f32dfd4fc099c78ce5dc0e91c4'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:56.839056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1416/2822] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:26:56 2022] DELETE /identity/v3/users/f952c1f32dfd4fc099c78ce5dc0e91c4 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:26:57.147638 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:57.149163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:57.165952 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8262ac06-4b0d-4671-97eb-7ef25bd15966 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:57.241883 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f50773d-ea1d-494e-b4ca-de50163e0dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:57.242555 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f50773d-ea1d-494e-b4ca-de50163e0dd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:57.242833 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f50773d-ea1d-494e-b4ca-de50163e0dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:57.243114 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f50773d-ea1d-494e-b4ca-de50163e0dd6 admin admin] PATH_INFO: `/v3/projects/6f5a32f178a04b4e8c2d8f9670322a97` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:57.244621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f50773d-ea1d-494e-b4ca-de50163e0dd6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f5a32f178a04b4e8c2d8f9670322a97)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:57.246589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f50773d-ea1d-494e-b4ca-de50163e0dd6 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:57.272312 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:57.302744 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6f50773d-ea1d-494e-b4ca-de50163e0dd6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f5a32f178a04b4e8c2d8f9670322a97', 'request_id': 'req-6f50773d-ea1d-494e-b4ca-de50163e0dd6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:57.309554 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6f50773d-ea1d-494e-b4ca-de50163e0dd6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f5a32f178a04b4e8c2d8f9670322a97', 'request_id': 'req-6f50773d-ea1d-494e-b4ca-de50163e0dd6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:57.318603 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6f50773d-ea1d-494e-b4ca-de50163e0dd6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f5a32f178a04b4e8c2d8f9670322a97', 'request_id': 'req-6f50773d-ea1d-494e-b4ca-de50163e0dd6'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:57.330540 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6f50773d-ea1d-494e-b4ca-de50163e0dd6 admin admin] The token cache is being invalidate because project 6f5a32f178a04b4e8c2d8f9670322a97 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:26:57.330831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6f50773d-ea1d-494e-b4ca-de50163e0dd6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:57.331577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-6f50773d-ea1d-494e-b4ca-de50163e0dd6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:26:57.333256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1407/2823] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:26:57 2022] DELETE /identity/v3/projects/6f5a32f178a04b4e8c2d8f9670322a97 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:26:57.339802 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b683ed34-8420-4383-93df-f45adfe21735 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:57.340107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b683ed34-8420-4383-93df-f45adfe21735 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:57.340363 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b683ed34-8420-4383-93df-f45adfe21735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:57.393998 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-b683ed34-8420-4383-93df-f45adfe21735 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:26:57.395145 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-b683ed34-8420-4383-93df-f45adfe21735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:57.449917 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:57.462208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1417/2824] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:26:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:26:57.471039 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acbc6f7-e235-4fc4-b422-9187e3ba1de9 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:57.471325 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acbc6f7-e235-4fc4-b422-9187e3ba1de9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:57.471620 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acbc6f7-e235-4fc4-b422-9187e3ba1de9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:57.492599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-1acbc6f7-e235-4fc4-b422-9187e3ba1de9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:26:57.493803 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-1acbc6f7-e235-4fc4-b422-9187e3ba1de9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:57.501915 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1408/2825] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:26:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:26:57.506311 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b683ed34-8420-4383-93df-f45adfe21735 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:57.516768 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b683ed34-8420-4383-93df-f45adfe21735 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:57.522624 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff2d7b0-622b-451d-ba28-bc9e1bc81bf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:57.523533 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff2d7b0-622b-451d-ba28-bc9e1bc81bf9 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:57.523873 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff2d7b0-622b-451d-ba28-bc9e1bc81bf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:57.524206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff2d7b0-622b-451d-ba28-bc9e1bc81bf9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:57.525057 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff2d7b0-622b-451d-ba28-bc9e1bc81bf9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:26:57.525457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff2d7b0-622b-451d-ba28-bc9e1bc81bf9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:57.527272 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff2d7b0-622b-451d-ba28-bc9e1bc81bf9 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:57.541160 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1418/2826] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:26:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:26:57.545743 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acbc6f7-e235-4fc4-b422-9187e3ba1de9 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:57.555874 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acbc6f7-e235-4fc4-b422-9187e3ba1de9 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:57.560879 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbcb2ee-dcc7-4b5f-b1ef-740b0ab4d78a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:57.561509 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbcb2ee-dcc7-4b5f-b1ef-740b0ab4d78a admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:57.561787 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbcb2ee-dcc7-4b5f-b1ef-740b0ab4d78a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:57.562058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbcb2ee-dcc7-4b5f-b1ef-740b0ab4d78a admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:57.562735 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbcb2ee-dcc7-4b5f-b1ef-740b0ab4d78a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:57.564960 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbcb2ee-dcc7-4b5f-b1ef-740b0ab4d78a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:57.594925 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1409/2827] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:26:57 2022] POST /identity/v3/projects => generated 409 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:26:57.600232 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff2d7b0-622b-451d-ba28-bc9e1bc81bf9 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:57.611172 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff2d7b0-622b-451d-ba28-bc9e1bc81bf9 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:57.675452 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295eb2b2-a6d9-40c7-9f4e-4f2e583aed0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:57.676186 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295eb2b2-a6d9-40c7-9f4e-4f2e583aed0b admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:57.676466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295eb2b2-a6d9-40c7-9f4e-4f2e583aed0b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:57.676911 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295eb2b2-a6d9-40c7-9f4e-4f2e583aed0b admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:57.677633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295eb2b2-a6d9-40c7-9f4e-4f2e583aed0b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:57.680114 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295eb2b2-a6d9-40c7-9f4e-4f2e583aed0b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:57.709107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1419/2828] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:26:57 2022] POST /identity/v3/users => generated 305 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:26:57.714033 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbcb2ee-dcc7-4b5f-b1ef-740b0ab4d78a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:57.725045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbcb2ee-dcc7-4b5f-b1ef-740b0ab4d78a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:57.730963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901eb05f-0ce3-4747-aae3-8cf24426909a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:57.731626 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901eb05f-0ce3-4747-aae3-8cf24426909a admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:57.731903 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901eb05f-0ce3-4747-aae3-8cf24426909a admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:57.732191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901eb05f-0ce3-4747-aae3-8cf24426909a admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:57.732898 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901eb05f-0ce3-4747-aae3-8cf24426909a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:57.734537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901eb05f-0ce3-4747-aae3-8cf24426909a admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:57.745904 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1410/2829] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:26:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:26:57.750400 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295eb2b2-a6d9-40c7-9f4e-4f2e583aed0b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:57.761908 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295eb2b2-a6d9-40c7-9f4e-4f2e583aed0b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:57.767239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7161443d-1ee9-4f50-b4b8-1afd74a81c5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:57.767936 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7161443d-1ee9-4f50-b4b8-1afd74a81c5a admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:57.768220 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7161443d-1ee9-4f50-b4b8-1afd74a81c5a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:57.768498 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7161443d-1ee9-4f50-b4b8-1afd74a81c5a admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:57.769204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7161443d-1ee9-4f50-b4b8-1afd74a81c5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:57.771484 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7161443d-1ee9-4f50-b4b8-1afd74a81c5a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:57.783672 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1420/2830] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:26:57 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:26:57.790702 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901eb05f-0ce3-4747-aae3-8cf24426909a admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:57.802118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901eb05f-0ce3-4747-aae3-8cf24426909a admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:57.807841 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7686d2ce-91bc-41ba-b991-5ca3b6e5a42e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:57.808542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7686d2ce-91bc-41ba-b991-5ca3b6e5a42e admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:57.808825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7686d2ce-91bc-41ba-b991-5ca3b6e5a42e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:57.809100 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7686d2ce-91bc-41ba-b991-5ca3b6e5a42e admin admin] PATH_INFO: `/v3/projects/417e649a3eff4115950a1779cb8668d5/users/520792088db44b1ba7ef67ae10bf38fd/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:57.829124 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7686d2ce-91bc-41ba-b991-5ca3b6e5a42e admin admin] RBAC: Authorizing `identity:create_grant(project_id=417e649a3eff4115950a1779cb8668d5, user_id=520792088db44b1ba7ef67ae10bf38fd, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:57.831875 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7686d2ce-91bc-41ba-b991-5ca3b6e5a42e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:57.848322 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1411/2831] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:26:57 2022] PUT /identity/v3/projects/417e649a3eff4115950a1779cb8668d5/users/520792088db44b1ba7ef67ae10bf38fd/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:26:57.868262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527175b4-6684-461f-8f9f-fed75dec47d8 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:57.869142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527175b4-6684-461f-8f9f-fed75dec47d8 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:57.869142 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527175b4-6684-461f-8f9f-fed75dec47d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:57.905234 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-527175b4-6684-461f-8f9f-fed75dec47d8 None None] MFA Rules not processed for user `520792088db44b1ba7ef67ae10bf38fd`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:26:57.906496 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-527175b4-6684-461f-8f9f-fed75dec47d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:57.991050 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1421/2832] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:26:57 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:26:57.995808 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7686d2ce-91bc-41ba-b991-5ca3b6e5a42e admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:58.073530 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7686d2ce-91bc-41ba-b991-5ca3b6e5a42e admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:58.078918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b8d3b8-766f-472b-a275-dad455d28b09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:58.079589 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b8d3b8-766f-472b-a275-dad455d28b09 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:58.079867 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b8d3b8-766f-472b-a275-dad455d28b09 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:58.080136 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b8d3b8-766f-472b-a275-dad455d28b09 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:58.080864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b8d3b8-766f-472b-a275-dad455d28b09 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:58.082836 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b8d3b8-766f-472b-a275-dad455d28b09 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:58.117107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1412/2833] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:26:57 2022] POST /identity/v3/projects => generated 409 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:26:58.118833 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527175b4-6684-461f-8f9f-fed75dec47d8 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:58.131613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527175b4-6684-461f-8f9f-fed75dec47d8 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:58.188023 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:58.200432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8338652-c7a7-4743-a5ee-3c35ec81e80b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:58.201059 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8338652-c7a7-4743-a5ee-3c35ec81e80b admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:58.201336 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8338652-c7a7-4743-a5ee-3c35ec81e80b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:58.201604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8338652-c7a7-4743-a5ee-3c35ec81e80b admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:58.202281 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8338652-c7a7-4743-a5ee-3c35ec81e80b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:58.204383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8338652-c7a7-4743-a5ee-3c35ec81e80b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:58.228853 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1422/2834] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:26:58 2022] POST /identity/v3/users => generated 311 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:26:58.233527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b8d3b8-766f-472b-a275-dad455d28b09 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:58.245177 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b8d3b8-766f-472b-a275-dad455d28b09 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:58.249930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836322af-00c5-4744-88f1-a2084352a45f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:58.250573 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836322af-00c5-4744-88f1-a2084352a45f admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:58.250850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836322af-00c5-4744-88f1-a2084352a45f admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:58.251113 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836322af-00c5-4744-88f1-a2084352a45f admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:58.251850 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836322af-00c5-4744-88f1-a2084352a45f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:58.253483 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836322af-00c5-4744-88f1-a2084352a45f admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:58.265304 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1413/2835] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:26:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:26:58.270021 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8338652-c7a7-4743-a5ee-3c35ec81e80b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:58.281138 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8338652-c7a7-4743-a5ee-3c35ec81e80b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:58.286056 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cecb48-1026-4435-8a03-215514f6af2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:58.287305 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cecb48-1026-4435-8a03-215514f6af2b admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:58.287617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cecb48-1026-4435-8a03-215514f6af2b admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:58.287888 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cecb48-1026-4435-8a03-215514f6af2b admin admin] PATH_INFO: `/v3/projects/1f73e9bd4d924f569fe532b738d2b6dd/users/f1e219feb56b42cfbb611013ba10c41c/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:58.304680 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cecb48-1026-4435-8a03-215514f6af2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f73e9bd4d924f569fe532b738d2b6dd, user_id=f1e219feb56b42cfbb611013ba10c41c, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:58.307612 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cecb48-1026-4435-8a03-215514f6af2b admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:58.325431 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1423/2836] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:26:58 2022] PUT /identity/v3/projects/1f73e9bd4d924f569fe532b738d2b6dd/users/f1e219feb56b42cfbb611013ba10c41c/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:26:58.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836322af-00c5-4744-88f1-a2084352a45f admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:58.341362 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836322af-00c5-4744-88f1-a2084352a45f admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:58.357200 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:26:58.412911 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c9fcf6-f3ab-4cc7-aaa0-3d9551ce322b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:58.413538 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c9fcf6-f3ab-4cc7-aaa0-3d9551ce322b admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:58.413812 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c9fcf6-f3ab-4cc7-aaa0-3d9551ce322b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:58.414080 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c9fcf6-f3ab-4cc7-aaa0-3d9551ce322b admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:58.414783 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c9fcf6-f3ab-4cc7-aaa0-3d9551ce322b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:58.416433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c9fcf6-f3ab-4cc7-aaa0-3d9551ce322b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:58.428979 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1414/2837] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:26:58 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:26:58.433733 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cecb48-1026-4435-8a03-215514f6af2b admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:58.446189 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cecb48-1026-4435-8a03-215514f6af2b admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:58.451224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55c17f1-90d3-474b-adda-31ecc0d110d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:58.451977 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55c17f1-90d3-474b-adda-31ecc0d110d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:58.452256 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55c17f1-90d3-474b-adda-31ecc0d110d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:58.452529 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55c17f1-90d3-474b-adda-31ecc0d110d2 admin admin] PATH_INFO: `/v3/projects/1f73e9bd4d924f569fe532b738d2b6dd/users/f1e219feb56b42cfbb611013ba10c41c/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:58.454818 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55c17f1-90d3-474b-adda-31ecc0d110d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f73e9bd4d924f569fe532b738d2b6dd, user_id=f1e219feb56b42cfbb611013ba10c41c, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:58.457542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55c17f1-90d3-474b-adda-31ecc0d110d2 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:58.470724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1424/2838] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:26:58 2022] PUT /identity/v3/projects/1f73e9bd4d924f569fe532b738d2b6dd/users/f1e219feb56b42cfbb611013ba10c41c/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:26:58.542582 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2344bcbc-47db-443d-a069-fd3724d89213 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:58.542887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2344bcbc-47db-443d-a069-fd3724d89213 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:58.543091 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2344bcbc-47db-443d-a069-fd3724d89213 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:58.575938 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-2344bcbc-47db-443d-a069-fd3724d89213 None None] MFA Rules not processed for user `f1e219feb56b42cfbb611013ba10c41c`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:26:58.577244 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-2344bcbc-47db-443d-a069-fd3724d89213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:58.667575 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1415/2839] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:26:58 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:26:58.682084 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55c17f1-90d3-474b-adda-31ecc0d110d2 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:58.684233 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e55c17f1-90d3-474b-adda-31ecc0d110d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:26:58.751824 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55c17f1-90d3-474b-adda-31ecc0d110d2 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:58.756298 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d8cb29-68f9-4b41-bf3a-cea76309b200 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:58.756937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d8cb29-68f9-4b41-bf3a-cea76309b200 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:58.757215 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d8cb29-68f9-4b41-bf3a-cea76309b200 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:58.757488 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d8cb29-68f9-4b41-bf3a-cea76309b200 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:58.774775 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d8cb29-68f9-4b41-bf3a-cea76309b200 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:58.776738 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d8cb29-68f9-4b41-bf3a-cea76309b200 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:58.788407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1425/2840] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:26:58 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:26:58.988127 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2344bcbc-47db-443d-a069-fd3724d89213 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:26:59.000154 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2344bcbc-47db-443d-a069-fd3724d89213 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:26:59.004034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844f2979-88cf-495b-a320-364cef93ea1d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:26:59.004714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f2979-88cf-495b-a320-364cef93ea1d service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:26:59.005060 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f2979-88cf-495b-a320-364cef93ea1d service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:26:59.005936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f2979-88cf-495b-a320-364cef93ea1d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:26:59.090017 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844f2979-88cf-495b-a320-364cef93ea1d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:26:59.092761 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844f2979-88cf-495b-a320-364cef93ea1d service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:26:59.111936 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1416/2841] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:26:58 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:27:07.874752 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d8cb29-68f9-4b41-bf3a-cea76309b200 service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:07.949459 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d8cb29-68f9-4b41-bf3a-cea76309b200 service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:07.954782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:07.955536 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:07.955865 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:07.956187 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] PATH_INFO: `/v3/users/520792088db44b1ba7ef67ae10bf38fd` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:07.957587 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] RBAC: Authorizing `identity:delete_user(user_id=520792088db44b1ba7ef67ae10bf38fd)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:07.960239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:08.020117 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '520792088db44b1ba7ef67ae10bf38fd'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:08.025005 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:08.038383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '520792088db44b1ba7ef67ae10bf38fd'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:08.040053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '520792088db44b1ba7ef67ae10bf38fd'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:08.065746 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '520792088db44b1ba7ef67ae10bf38fd'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:08.066900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '520792088db44b1ba7ef67ae10bf38fd'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:08.088313 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1426/2842] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:27:07 2022] DELETE /identity/v3/users/520792088db44b1ba7ef67ae10bf38fd => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:08.093698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28584e6-ff1d-4df3-aa57-1f66a75df1b2 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:08.094349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28584e6-ff1d-4df3-aa57-1f66a75df1b2 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:08.094349 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28584e6-ff1d-4df3-aa57-1f66a75df1b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:08.114227 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-c28584e6-ff1d-4df3-aa57-1f66a75df1b2 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:08.115524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c28584e6-ff1d-4df3-aa57-1f66a75df1b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:08.151895 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:08.180941 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1417/2843] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:08.192248 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:08.194971 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:08.283555 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e737a66b-187f-4829-821a-6927a11d407e admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:08.287328 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef21c65-cb3e-4e87-9044-58248c4d13bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:08.287995 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef21c65-cb3e-4e87-9044-58248c4d13bf service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:08.288277 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef21c65-cb3e-4e87-9044-58248c4d13bf service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:08.288555 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef21c65-cb3e-4e87-9044-58248c4d13bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:08.298462 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef21c65-cb3e-4e87-9044-58248c4d13bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:08.299955 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef21c65-cb3e-4e87-9044-58248c4d13bf service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:08.312558 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1427/2844] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:27:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:27:08.657207 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28584e6-ff1d-4df3-aa57-1f66a75df1b2 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:08.658904 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c28584e6-ff1d-4df3-aa57-1f66a75df1b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:08.679964 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28584e6-ff1d-4df3-aa57-1f66a75df1b2 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:08.686778 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:08.688336 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:08.688882 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:08.689330 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] PATH_INFO: `/v3/projects/417e649a3eff4115950a1779cb8668d5` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:08.691818 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=417e649a3eff4115950a1779cb8668d5)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:08.695689 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:08.755245 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '417e649a3eff4115950a1779cb8668d5', 'request_id': 'req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:08.763204 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '417e649a3eff4115950a1779cb8668d5', 'request_id': 'req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:08.773498 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '417e649a3eff4115950a1779cb8668d5', 'request_id': 'req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:08.785253 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] The token cache is being invalidate because project 417e649a3eff4115950a1779cb8668d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:27:08.785557 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:08.786767 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:08.788354 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1418/2845] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:27:08 2022] DELETE /identity/v3/projects/417e649a3eff4115950a1779cb8668d5 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:27:08.794606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef21c65-cb3e-4e87-9044-58248c4d13bf service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:08.812290 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef21c65-cb3e-4e87-9044-58248c4d13bf service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:08.909785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f4f890-f31d-462d-b5bd-752844b52d1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:08.910646 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4f890-f31d-462d-b5bd-752844b52d1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:08.910785 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4f890-f31d-462d-b5bd-752844b52d1a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:08.911165 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4f890-f31d-462d-b5bd-752844b52d1a admin admin] PATH_INFO: `/v3/users/f1e219feb56b42cfbb611013ba10c41c` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:08.913271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f4f890-f31d-462d-b5bd-752844b52d1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=f1e219feb56b42cfbb611013ba10c41c)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:08.915255 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f4f890-f31d-462d-b5bd-752844b52d1a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:08.991070 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a4f4f890-f31d-462d-b5bd-752844b52d1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f1e219feb56b42cfbb611013ba10c41c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:08.998806 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a4f4f890-f31d-462d-b5bd-752844b52d1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f1e219feb56b42cfbb611013ba10c41c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:08.999632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a4f4f890-f31d-462d-b5bd-752844b52d1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f1e219feb56b42cfbb611013ba10c41c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:09.019381 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:09.022797 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a4f4f890-f31d-462d-b5bd-752844b52d1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f1e219feb56b42cfbb611013ba10c41c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:09.024339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-a4f4f890-f31d-462d-b5bd-752844b52d1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f1e219feb56b42cfbb611013ba10c41c'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:09.042185 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1428/2846] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:27:08 2022] DELETE /identity/v3/users/f1e219feb56b42cfbb611013ba10c41c => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:09.373819 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:09.375487 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:09.406416 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24396a2-1cc7-4a02-97b4-03b68ddf5ed5 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:09.476896 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d74bf53-c8e2-4109-b868-9e048f78a2cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:09.477641 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d74bf53-c8e2-4109-b868-9e048f78a2cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:09.477923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d74bf53-c8e2-4109-b868-9e048f78a2cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:09.478266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d74bf53-c8e2-4109-b868-9e048f78a2cf admin admin] PATH_INFO: `/v3/projects/1f73e9bd4d924f569fe532b738d2b6dd` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:09.479864 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d74bf53-c8e2-4109-b868-9e048f78a2cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=1f73e9bd4d924f569fe532b738d2b6dd)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:09.482314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d74bf53-c8e2-4109-b868-9e048f78a2cf admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:09.535251 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2d74bf53-c8e2-4109-b868-9e048f78a2cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1f73e9bd4d924f569fe532b738d2b6dd', 'request_id': 'req-2d74bf53-c8e2-4109-b868-9e048f78a2cf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:09.541934 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2d74bf53-c8e2-4109-b868-9e048f78a2cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1f73e9bd4d924f569fe532b738d2b6dd', 'request_id': 'req-2d74bf53-c8e2-4109-b868-9e048f78a2cf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:09.550452 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2d74bf53-c8e2-4109-b868-9e048f78a2cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1f73e9bd4d924f569fe532b738d2b6dd', 'request_id': 'req-2d74bf53-c8e2-4109-b868-9e048f78a2cf'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:09.553786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:09.568390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2d74bf53-c8e2-4109-b868-9e048f78a2cf admin admin] The token cache is being invalidate because project 1f73e9bd4d924f569fe532b738d2b6dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:27:09.568829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2d74bf53-c8e2-4109-b868-9e048f78a2cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:09.569427 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-2d74bf53-c8e2-4109-b868-9e048f78a2cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:09.571122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1419/2847] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:27:09 2022] DELETE /identity/v3/projects/1f73e9bd4d924f569fe532b738d2b6dd => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:27:09.576280 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd5c33a-401d-4069-8bd9-21d45c8698ce None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:09.576606 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd5c33a-401d-4069-8bd9-21d45c8698ce None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:09.576928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd5c33a-401d-4069-8bd9-21d45c8698ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:09.598199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-edd5c33a-401d-4069-8bd9-21d45c8698ce None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:09.599496 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-edd5c33a-401d-4069-8bd9-21d45c8698ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:09.668089 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1429/2848] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:09.676166 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1e903f-ab4b-4df6-bd5b-b1c23c7569e4 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:09.676513 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1e903f-ab4b-4df6-bd5b-b1c23c7569e4 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:09.677348 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1e903f-ab4b-4df6-bd5b-b1c23c7569e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:09.700974 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-9d1e903f-ab4b-4df6-bd5b-b1c23c7569e4 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:09.702288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-9d1e903f-ab4b-4df6-bd5b-b1c23c7569e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:09.711234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1420/2849] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:27:09.715907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd5c33a-401d-4069-8bd9-21d45c8698ce None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:09.726931 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd5c33a-401d-4069-8bd9-21d45c8698ce None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:09.733989 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02755912-7f07-4bef-bbbc-5a4df1dfcc81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:09.735095 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02755912-7f07-4bef-bbbc-5a4df1dfcc81 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:09.735632 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02755912-7f07-4bef-bbbc-5a4df1dfcc81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:09.736122 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02755912-7f07-4bef-bbbc-5a4df1dfcc81 admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:09.737500 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02755912-7f07-4bef-bbbc-5a4df1dfcc81 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:27:09.738045 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02755912-7f07-4bef-bbbc-5a4df1dfcc81 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:09.741756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02755912-7f07-4bef-bbbc-5a4df1dfcc81 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:09.764463 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1430/2850] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:27:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:27:09.769132 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1e903f-ab4b-4df6-bd5b-b1c23c7569e4 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:09.779122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1e903f-ab4b-4df6-bd5b-b1c23c7569e4 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:09.784194 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369283e8-52c8-4c08-a142-ab005e52f786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:09.785003 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369283e8-52c8-4c08-a142-ab005e52f786 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:09.785401 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369283e8-52c8-4c08-a142-ab005e52f786 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:09.785789 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369283e8-52c8-4c08-a142-ab005e52f786 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:09.787103 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369283e8-52c8-4c08-a142-ab005e52f786 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:09.789390 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369283e8-52c8-4c08-a142-ab005e52f786 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:09.826495 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1421/2851] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:27:09 2022] POST /identity/v3/projects => generated 383 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:27:09.828585 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02755912-7f07-4bef-bbbc-5a4df1dfcc81 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:09.839565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02755912-7f07-4bef-bbbc-5a4df1dfcc81 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:09.908808 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f9aee5-cd6c-49eb-bb22-93944a15903d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:09.909476 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f9aee5-cd6c-49eb-bb22-93944a15903d admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:09.909760 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f9aee5-cd6c-49eb-bb22-93944a15903d admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:09.910112 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f9aee5-cd6c-49eb-bb22-93944a15903d admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:09.910790 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f9aee5-cd6c-49eb-bb22-93944a15903d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:09.913314 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f9aee5-cd6c-49eb-bb22-93944a15903d admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:09.937409 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1431/2852] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:27:09 2022] POST /identity/v3/users => generated 292 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:27:09.942377 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369283e8-52c8-4c08-a142-ab005e52f786 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:09.953661 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369283e8-52c8-4c08-a142-ab005e52f786 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:09.958784 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f532ea1e-e86d-489f-ae8a-d28b91118156 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:09.959447 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f532ea1e-e86d-489f-ae8a-d28b91118156 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:09.959723 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f532ea1e-e86d-489f-ae8a-d28b91118156 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:09.959987 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f532ea1e-e86d-489f-ae8a-d28b91118156 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:09.960684 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f532ea1e-e86d-489f-ae8a-d28b91118156 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:09.962320 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f532ea1e-e86d-489f-ae8a-d28b91118156 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:09.974586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1422/2853] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:27:09 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:27:09.979150 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f9aee5-cd6c-49eb-bb22-93944a15903d admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:09.990683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f9aee5-cd6c-49eb-bb22-93944a15903d admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:09.995435 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcfa6a6-6024-4227-9171-8bd63605cb31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:09.996063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcfa6a6-6024-4227-9171-8bd63605cb31 admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:09.996339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcfa6a6-6024-4227-9171-8bd63605cb31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:09.996607 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcfa6a6-6024-4227-9171-8bd63605cb31 admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:09.997304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcfa6a6-6024-4227-9171-8bd63605cb31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:09.998953 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcfa6a6-6024-4227-9171-8bd63605cb31 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:10.011432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1432/2854] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:27:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:27:10.015207 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f532ea1e-e86d-489f-ae8a-d28b91118156 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:10.026010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f532ea1e-e86d-489f-ae8a-d28b91118156 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:10.031102 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e815097d-43ed-4c1c-96b3-b1a30164a2e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:10.031831 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e815097d-43ed-4c1c-96b3-b1a30164a2e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:10.032314 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e815097d-43ed-4c1c-96b3-b1a30164a2e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:10.032595 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e815097d-43ed-4c1c-96b3-b1a30164a2e5 admin admin] PATH_INFO: `/v3/projects/868cb9a2a1314a5397c305542fa8a8ed/users/10d98a82e642485f9269ad8f5914cce3/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:10.049767 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e815097d-43ed-4c1c-96b3-b1a30164a2e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=868cb9a2a1314a5397c305542fa8a8ed, user_id=10d98a82e642485f9269ad8f5914cce3, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:10.053139 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e815097d-43ed-4c1c-96b3-b1a30164a2e5 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:10.073531 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1423/2855] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:27:10 2022] PUT /identity/v3/projects/868cb9a2a1314a5397c305542fa8a8ed/users/10d98a82e642485f9269ad8f5914cce3/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:10.086460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdde3c-3fc4-491c-bed7-8494661dbe1d None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:10.086744 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdde3c-3fc4-491c-bed7-8494661dbe1d None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:10.087198 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdde3c-3fc4-491c-bed7-8494661dbe1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:10.119639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-7ffdde3c-3fc4-491c-bed7-8494661dbe1d None None] MFA Rules not processed for user `10d98a82e642485f9269ad8f5914cce3`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:10.120881 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-7ffdde3c-3fc4-491c-bed7-8494661dbe1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:10.170556 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:10.196798 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1433/2856] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:10 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:10.201734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e815097d-43ed-4c1c-96b3-b1a30164a2e5 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:10.273248 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e815097d-43ed-4c1c-96b3-b1a30164a2e5 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:10.277984 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8c326b-4e26-4e44-b9a0-56e83297dc88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:10.278612 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8c326b-4e26-4e44-b9a0-56e83297dc88 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:10.278920 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8c326b-4e26-4e44-b9a0-56e83297dc88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:10.279190 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8c326b-4e26-4e44-b9a0-56e83297dc88 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:10.279899 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8c326b-4e26-4e44-b9a0-56e83297dc88 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:10.281806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8c326b-4e26-4e44-b9a0-56e83297dc88 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:10.312025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1424/2857] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:27:10 2022] POST /identity/v3/projects => generated 383 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:27:10.316845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffdde3c-3fc4-491c-bed7-8494661dbe1d None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:10.329928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffdde3c-3fc4-491c-bed7-8494661dbe1d None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:10.400703 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdb1271-9710-4d66-a08a-79a333a06447 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:10.401658 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdb1271-9710-4d66-a08a-79a333a06447 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:10.402129 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdb1271-9710-4d66-a08a-79a333a06447 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:10.402691 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdb1271-9710-4d66-a08a-79a333a06447 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:10.403956 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdb1271-9710-4d66-a08a-79a333a06447 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:10.406065 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdb1271-9710-4d66-a08a-79a333a06447 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:10.431304 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1434/2858] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:27:10 2022] POST /identity/v3/users => generated 298 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:27:10.435676 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8c326b-4e26-4e44-b9a0-56e83297dc88 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:10.448786 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8c326b-4e26-4e44-b9a0-56e83297dc88 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:10.454019 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e5ac58-8bd8-4510-855a-5feff1b4d144 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:10.454658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e5ac58-8bd8-4510-855a-5feff1b4d144 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:10.454938 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e5ac58-8bd8-4510-855a-5feff1b4d144 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:10.455218 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e5ac58-8bd8-4510-855a-5feff1b4d144 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:10.455967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e5ac58-8bd8-4510-855a-5feff1b4d144 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:10.457586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e5ac58-8bd8-4510-855a-5feff1b4d144 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:10.470210 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1425/2859] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:27:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:27:10.475912 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdb1271-9710-4d66-a08a-79a333a06447 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:10.487300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdb1271-9710-4d66-a08a-79a333a06447 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:10.492604 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d20c7d2-e5ba-4759-a7b9-cbde7aec1a43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:10.493297 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d20c7d2-e5ba-4759-a7b9-cbde7aec1a43 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:10.493576 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d20c7d2-e5ba-4759-a7b9-cbde7aec1a43 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:10.493864 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d20c7d2-e5ba-4759-a7b9-cbde7aec1a43 admin admin] PATH_INFO: `/v3/projects/0733c3f7c26e479988994a6ff15e14e4/users/d6c61547a667459784b869641045f64d/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:10.512328 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d20c7d2-e5ba-4759-a7b9-cbde7aec1a43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0733c3f7c26e479988994a6ff15e14e4, user_id=d6c61547a667459784b869641045f64d, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:10.515118 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d20c7d2-e5ba-4759-a7b9-cbde7aec1a43 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:10.530970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1435/2860] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:27:10 2022] PUT /identity/v3/projects/0733c3f7c26e479988994a6ff15e14e4/users/d6c61547a667459784b869641045f64d/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:10.535273 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e5ac58-8bd8-4510-855a-5feff1b4d144 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:10.547903 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e5ac58-8bd8-4510-855a-5feff1b4d144 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:10.615344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c90ad3-1aea-406a-b5f9-4900a22364c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:10.616041 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c90ad3-1aea-406a-b5f9-4900a22364c2 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:10.616318 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c90ad3-1aea-406a-b5f9-4900a22364c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:10.616592 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c90ad3-1aea-406a-b5f9-4900a22364c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:10.617296 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c90ad3-1aea-406a-b5f9-4900a22364c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:10.618930 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c90ad3-1aea-406a-b5f9-4900a22364c2 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:10.621543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:10.632433 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1426/2861] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:27:10 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 31 01:27:10.637284 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d20c7d2-e5ba-4759-a7b9-cbde7aec1a43 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:10.649341 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d20c7d2-e5ba-4759-a7b9-cbde7aec1a43 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:10.654276 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265ae571-40cd-4074-ad9d-997af0df0f50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:10.655016 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265ae571-40cd-4074-ad9d-997af0df0f50 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:10.655383 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265ae571-40cd-4074-ad9d-997af0df0f50 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:10.655713 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265ae571-40cd-4074-ad9d-997af0df0f50 admin admin] PATH_INFO: `/v3/projects/0733c3f7c26e479988994a6ff15e14e4/users/d6c61547a667459784b869641045f64d/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:10.657938 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265ae571-40cd-4074-ad9d-997af0df0f50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0733c3f7c26e479988994a6ff15e14e4, user_id=d6c61547a667459784b869641045f64d, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:10.661684 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265ae571-40cd-4074-ad9d-997af0df0f50 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:10.674571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1436/2862] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:27:10 2022] PUT /identity/v3/projects/0733c3f7c26e479988994a6ff15e14e4/users/d6c61547a667459784b869641045f64d/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:10.686337 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08d6133-7cbf-4e6a-a914-ba92db4e340f None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:10.686675 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08d6133-7cbf-4e6a-a914-ba92db4e340f None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:10.686894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08d6133-7cbf-4e6a-a914-ba92db4e340f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:10.722193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-c08d6133-7cbf-4e6a-a914-ba92db4e340f None None] MFA Rules not processed for user `d6c61547a667459784b869641045f64d`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:10.723462 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c08d6133-7cbf-4e6a-a914-ba92db4e340f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:10.805421 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1427/2863] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:10 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:10.818267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265ae571-40cd-4074-ad9d-997af0df0f50 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:10.820349 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-265ae571-40cd-4074-ad9d-997af0df0f50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:10.919004 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265ae571-40cd-4074-ad9d-997af0df0f50 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:10.925208 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01de5a8a-13d8-49d6-8c19-7a241bf9cecc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:10.926274 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01de5a8a-13d8-49d6-8c19-7a241bf9cecc service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:10.926789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01de5a8a-13d8-49d6-8c19-7a241bf9cecc service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:10.927332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01de5a8a-13d8-49d6-8c19-7a241bf9cecc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:10.945276 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01de5a8a-13d8-49d6-8c19-7a241bf9cecc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:10.948825 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01de5a8a-13d8-49d6-8c19-7a241bf9cecc service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:10.969105 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1437/2864] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:27:10 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:27:11.051845 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08d6133-7cbf-4e6a-a914-ba92db4e340f None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:11.062983 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08d6133-7cbf-4e6a-a914-ba92db4e340f None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:11.067220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e6b47f-da90-4b19-b270-eed9ce8c297d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:11.067942 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e6b47f-da90-4b19-b270-eed9ce8c297d service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:11.068256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e6b47f-da90-4b19-b270-eed9ce8c297d service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:11.068565 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e6b47f-da90-4b19-b270-eed9ce8c297d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:11.169393 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e6b47f-da90-4b19-b270-eed9ce8c297d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:11.172028 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e6b47f-da90-4b19-b270-eed9ce8c297d service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:11.188956 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1428/2865] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:27:11 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:27:11.537418 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01de5a8a-13d8-49d6-8c19-7a241bf9cecc service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:11.588855 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:11.613590 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01de5a8a-13d8-49d6-8c19-7a241bf9cecc service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:11.618825 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:11.619527 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:11.619805 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:11.620101 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] PATH_INFO: `/v3/users/10d98a82e642485f9269ad8f5914cce3` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:11.621337 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] RBAC: Authorizing `identity:delete_user(user_id=10d98a82e642485f9269ad8f5914cce3)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:11.623307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:11.680008 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '10d98a82e642485f9269ad8f5914cce3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:11.688169 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '10d98a82e642485f9269ad8f5914cce3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:11.688900 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '10d98a82e642485f9269ad8f5914cce3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:11.702683 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '10d98a82e642485f9269ad8f5914cce3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:11.703449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '10d98a82e642485f9269ad8f5914cce3'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:11.719020 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1438/2866] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:27:11 2022] DELETE /identity/v3/users/10d98a82e642485f9269ad8f5914cce3 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:27:11.724054 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048d0033-45bf-4901-8dc4-4f8143a1e2a9 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:11.724676 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048d0033-45bf-4901-8dc4-4f8143a1e2a9 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:11.724676 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048d0033-45bf-4901-8dc4-4f8143a1e2a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:11.747995 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-048d0033-45bf-4901-8dc4-4f8143a1e2a9 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:11.749377 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-048d0033-45bf-4901-8dc4-4f8143a1e2a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:11.818770 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1429/2867] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:11.831450 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:11.833722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:11.940920 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debbb5e2-bfba-4d3c-aadb-fa382e4ec18f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:11.947630 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be98015-bb3b-44d7-81fb-5d5f97897956 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:11.948725 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be98015-bb3b-44d7-81fb-5d5f97897956 service neutron] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:11.949224 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be98015-bb3b-44d7-81fb-5d5f97897956 service neutron] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:11.949726 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be98015-bb3b-44d7-81fb-5d5f97897956 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:11.966773 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be98015-bb3b-44d7-81fb-5d5f97897956 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:11.969577 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be98015-bb3b-44d7-81fb-5d5f97897956 service neutron] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:11.988756 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1439/2868] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:27:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:27:12.361269 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048d0033-45bf-4901-8dc4-4f8143a1e2a9 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:12.363104 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-048d0033-45bf-4901-8dc4-4f8143a1e2a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:12.384458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048d0033-45bf-4901-8dc4-4f8143a1e2a9 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:12.389607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:12.390280 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:12.390551 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:12.391010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] PATH_INFO: `/v3/projects/868cb9a2a1314a5397c305542fa8a8ed` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:12.392394 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] RBAC: Authorizing `identity:delete_project(project_id=868cb9a2a1314a5397c305542fa8a8ed)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:12.394750 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:12.416857 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:12.464621 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '868cb9a2a1314a5397c305542fa8a8ed', 'request_id': 'req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:12.472045 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '868cb9a2a1314a5397c305542fa8a8ed', 'request_id': 'req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:12.480643 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '868cb9a2a1314a5397c305542fa8a8ed', 'request_id': 'req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:12.494397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] The token cache is being invalidate because project 868cb9a2a1314a5397c305542fa8a8ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:27:12.494784 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:12.495505 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:12.497073 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1430/2869] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:27:12 2022] DELETE /identity/v3/projects/868cb9a2a1314a5397c305542fa8a8ed => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:12.501259 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be98015-bb3b-44d7-81fb-5d5f97897956 service neutron] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:12.512581 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be98015-bb3b-44d7-81fb-5d5f97897956 service neutron] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:12.579667 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf98d4d-5852-4017-9300-0035ef7ccf93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:12.580315 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf98d4d-5852-4017-9300-0035ef7ccf93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:12.580591 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf98d4d-5852-4017-9300-0035ef7ccf93 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:12.580861 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf98d4d-5852-4017-9300-0035ef7ccf93 admin admin] PATH_INFO: `/v3/users/d6c61547a667459784b869641045f64d` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:12.582043 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf98d4d-5852-4017-9300-0035ef7ccf93 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6c61547a667459784b869641045f64d)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:12.584219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf98d4d-5852-4017-9300-0035ef7ccf93 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:12.587380 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:12.643599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4bf98d4d-5852-4017-9300-0035ef7ccf93 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6c61547a667459784b869641045f64d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:12.651949 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4bf98d4d-5852-4017-9300-0035ef7ccf93 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6c61547a667459784b869641045f64d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:12.652970 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4bf98d4d-5852-4017-9300-0035ef7ccf93 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6c61547a667459784b869641045f64d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:12.667731 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4bf98d4d-5852-4017-9300-0035ef7ccf93 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6c61547a667459784b869641045f64d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:12.668611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-4bf98d4d-5852-4017-9300-0035ef7ccf93 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6c61547a667459784b869641045f64d'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:12.684207 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1440/2870] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:27:12 2022] DELETE /identity/v3/users/d6c61547a667459784b869641045f64d => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:13.022291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:13.024059 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:13.044698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8144cb-fcf5-4aa4-b77e-fd243268fd40 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:13.122564 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:13.123226 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:13.123542 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:13.123817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e admin admin] PATH_INFO: `/v3/projects/0733c3f7c26e479988994a6ff15e14e4` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:13.125079 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=0733c3f7c26e479988994a6ff15e14e4)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:13.127036 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:13.194698 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0733c3f7c26e479988994a6ff15e14e4', 'request_id': 'req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:13.207270 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0733c3f7c26e479988994a6ff15e14e4', 'request_id': 'req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:13.221051 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0733c3f7c26e479988994a6ff15e14e4', 'request_id': 'req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:13.242130 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e admin admin] The token cache is being invalidate because project 0733c3f7c26e479988994a6ff15e14e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80209) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:27:13.242630 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:13.243763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-3cb6e0fa-d616-43f2-9302-bc6f4ef1fe7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:13.246413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1431/2871] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:27:13 2022] DELETE /identity/v3/projects/0733c3f7c26e479988994a6ff15e14e4 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:13.253102 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0486c2ca-06b0-4f59-9252-2d476f7b71af None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:13.253722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0486c2ca-06b0-4f59-9252-2d476f7b71af None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:13.253722 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0486c2ca-06b0-4f59-9252-2d476f7b71af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:13.276227 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-0486c2ca-06b0-4f59-9252-2d476f7b71af None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:13.277617 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-0486c2ca-06b0-4f59-9252-2d476f7b71af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:13.351853 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1441/2872] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:27:13.360331 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be0db41-f89c-498f-bfcd-29e21e68a5e5 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:13.360678 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be0db41-f89c-498f-bfcd-29e21e68a5e5 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:13.361071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be0db41-f89c-498f-bfcd-29e21e68a5e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:13.382579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-6be0db41-f89c-498f-bfcd-29e21e68a5e5 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:13.383888 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-6be0db41-f89c-498f-bfcd-29e21e68a5e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:13.392634 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1432/2873] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:13.397889 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0486c2ca-06b0-4f59-9252-2d476f7b71af None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:13.408974 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0486c2ca-06b0-4f59-9252-2d476f7b71af None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:13.414708 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492a8d26-b552-4c66-911a-36e6bb1f860d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:13.415378 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a8d26-b552-4c66-911a-36e6bb1f860d admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:13.415662 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a8d26-b552-4c66-911a-36e6bb1f860d admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:13.415937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a8d26-b552-4c66-911a-36e6bb1f860d admin admin] PATH_INFO: `/v3/domains` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:13.416651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492a8d26-b552-4c66-911a-36e6bb1f860d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80208) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:27:13.416957 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492a8d26-b552-4c66-911a-36e6bb1f860d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:13.418696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492a8d26-b552-4c66-911a-36e6bb1f860d admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:13.432232 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1442/2874] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:27:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 31 01:27:13.436841 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be0db41-f89c-498f-bfcd-29e21e68a5e5 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:13.447533 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be0db41-f89c-498f-bfcd-29e21e68a5e5 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:13.452438 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaac79db-c3ea-4a51-b758-af0c8fc6467b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:13.453076 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaac79db-c3ea-4a51-b758-af0c8fc6467b admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:13.453351 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaac79db-c3ea-4a51-b758-af0c8fc6467b admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:13.453624 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaac79db-c3ea-4a51-b758-af0c8fc6467b admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:13.454317 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaac79db-c3ea-4a51-b758-af0c8fc6467b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:13.456321 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaac79db-c3ea-4a51-b758-af0c8fc6467b admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:13.490137 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1433/2875] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:27:13 2022] POST /identity/v3/projects => generated 383 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:27:13.495115 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492a8d26-b552-4c66-911a-36e6bb1f860d admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:13.509570 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492a8d26-b552-4c66-911a-36e6bb1f860d admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:13.578372 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea93cb9a-ce7d-486a-bb24-40b14036b9c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:13.579002 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93cb9a-ce7d-486a-bb24-40b14036b9c7 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:13.579286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93cb9a-ce7d-486a-bb24-40b14036b9c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:13.579596 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93cb9a-ce7d-486a-bb24-40b14036b9c7 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:13.580269 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea93cb9a-ce7d-486a-bb24-40b14036b9c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:13.582239 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea93cb9a-ce7d-486a-bb24-40b14036b9c7 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:13.600600 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:13.609461 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1443/2876] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:27:13 2022] POST /identity/v3/users => generated 292 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:27:13.614249 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaac79db-c3ea-4a51-b758-af0c8fc6467b admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:13.625673 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaac79db-c3ea-4a51-b758-af0c8fc6467b admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:13.631464 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07664b77-4d34-49b7-acda-717ae7715e63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:13.632105 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07664b77-4d34-49b7-acda-717ae7715e63 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:13.632381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07664b77-4d34-49b7-acda-717ae7715e63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:13.632654 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07664b77-4d34-49b7-acda-717ae7715e63 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:13.633363 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07664b77-4d34-49b7-acda-717ae7715e63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:13.635007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07664b77-4d34-49b7-acda-717ae7715e63 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:13.636666 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:13.654765 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1434/2877] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:27:13 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:27:13.659751 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea93cb9a-ce7d-486a-bb24-40b14036b9c7 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:13.679677 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea93cb9a-ce7d-486a-bb24-40b14036b9c7 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:13.688636 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcb77ec-2b1f-435f-bbbe-7e96d8132a8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:13.690204 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcb77ec-2b1f-435f-bbbe-7e96d8132a8a admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:13.690769 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcb77ec-2b1f-435f-bbbe-7e96d8132a8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:13.691407 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcb77ec-2b1f-435f-bbbe-7e96d8132a8a admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:13.693384 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcb77ec-2b1f-435f-bbbe-7e96d8132a8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:13.696221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcb77ec-2b1f-435f-bbbe-7e96d8132a8a admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:13.718086 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1444/2878] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:27:13 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:27:13.723057 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07664b77-4d34-49b7-acda-717ae7715e63 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:13.733873 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07664b77-4d34-49b7-acda-717ae7715e63 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:13.738792 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f38abb-c496-43cd-8014-5018d872edc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:13.739521 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f38abb-c496-43cd-8014-5018d872edc9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:13.739795 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f38abb-c496-43cd-8014-5018d872edc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:13.740069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f38abb-c496-43cd-8014-5018d872edc9 admin admin] PATH_INFO: `/v3/projects/f0a560fca7d54959afb8cc66655624dc/users/def7b60ff1d440f1bfc74c43da96e928/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:13.757806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f38abb-c496-43cd-8014-5018d872edc9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0a560fca7d54959afb8cc66655624dc, user_id=def7b60ff1d440f1bfc74c43da96e928, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:13.760734 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f38abb-c496-43cd-8014-5018d872edc9 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:13.777341 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1435/2879] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:27:13 2022] PUT /identity/v3/projects/f0a560fca7d54959afb8cc66655624dc/users/def7b60ff1d440f1bfc74c43da96e928/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:13.790058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6d4f8f-e92f-40ce-8d0e-9891efcff440 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:13.790502 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6d4f8f-e92f-40ce-8d0e-9891efcff440 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:13.790813 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6d4f8f-e92f-40ce-8d0e-9891efcff440 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:13.826063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-fb6d4f8f-e92f-40ce-8d0e-9891efcff440 None None] MFA Rules not processed for user `def7b60ff1d440f1bfc74c43da96e928`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:13.827470 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-fb6d4f8f-e92f-40ce-8d0e-9891efcff440 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:13.908639 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1445/2880] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:13 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 31 01:27:13.913736 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f38abb-c496-43cd-8014-5018d872edc9 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:13.993793 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f38abb-c496-43cd-8014-5018d872edc9 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:13.998828 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97910deb-d1bb-4a3a-8015-c2887cae24ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:13.999579 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97910deb-d1bb-4a3a-8015-c2887cae24ad admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:13.999909 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97910deb-d1bb-4a3a-8015-c2887cae24ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.000261 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97910deb-d1bb-4a3a-8015-c2887cae24ad admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:14.001030 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97910deb-d1bb-4a3a-8015-c2887cae24ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:14.003034 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97910deb-d1bb-4a3a-8015-c2887cae24ad admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:14.037256 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1436/2881] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:27:13 2022] POST /identity/v3/projects => generated 385 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:27:14.041599 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6d4f8f-e92f-40ce-8d0e-9891efcff440 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:14.053845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6d4f8f-e92f-40ce-8d0e-9891efcff440 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:14.123438 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80282d3c-6ce3-446e-ad6b-768de16a5584 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:14.124185 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80282d3c-6ce3-446e-ad6b-768de16a5584 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:14.124509 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80282d3c-6ce3-446e-ad6b-768de16a5584 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.124879 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80282d3c-6ce3-446e-ad6b-768de16a5584 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:14.125626 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80282d3c-6ce3-446e-ad6b-768de16a5584 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:14.128294 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80282d3c-6ce3-446e-ad6b-768de16a5584 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:14.159579 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1446/2882] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:27:14 2022] POST /identity/v3/users => generated 293 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:27:14.163957 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97910deb-d1bb-4a3a-8015-c2887cae24ad admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:14.175986 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97910deb-d1bb-4a3a-8015-c2887cae24ad admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:14.181527 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcfa847-6f09-42fa-893d-1ee871779a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:14.182160 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcfa847-6f09-42fa-893d-1ee871779a03 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:14.182440 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcfa847-6f09-42fa-893d-1ee871779a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.182714 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcfa847-6f09-42fa-893d-1ee871779a03 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:14.183456 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcfa847-6f09-42fa-893d-1ee871779a03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:14.185098 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcfa847-6f09-42fa-893d-1ee871779a03 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:14.198658 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1437/2883] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:27:14 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:27:14.203089 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80282d3c-6ce3-446e-ad6b-768de16a5584 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:14.214651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80282d3c-6ce3-446e-ad6b-768de16a5584 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:14.220232 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bbbc58-f832-47e0-83e7-d597d494675c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:14.221005 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bbbc58-f832-47e0-83e7-d597d494675c admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:14.221376 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bbbc58-f832-47e0-83e7-d597d494675c admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.221741 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bbbc58-f832-47e0-83e7-d597d494675c admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:14.222538 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29bbbc58-f832-47e0-83e7-d597d494675c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:14.224937 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29bbbc58-f832-47e0-83e7-d597d494675c admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:14.237434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1447/2884] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:27:14 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:27:14.242842 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcfa847-6f09-42fa-893d-1ee871779a03 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:14.254718 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcfa847-6f09-42fa-893d-1ee871779a03 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:14.259806 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bf2906-d84a-41e3-9ac5-de7acc5af161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:14.260502 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bf2906-d84a-41e3-9ac5-de7acc5af161 admin admin] REQUEST_METHOD: `PUT` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:14.260776 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bf2906-d84a-41e3-9ac5-de7acc5af161 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.261047 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bf2906-d84a-41e3-9ac5-de7acc5af161 admin admin] PATH_INFO: `/v3/projects/9392602c02724740bef91b490d074c3c/users/8ffbd6a35f2d4b3192ae75fe7cd9092b/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:14.278211 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bf2906-d84a-41e3-9ac5-de7acc5af161 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9392602c02724740bef91b490d074c3c, user_id=8ffbd6a35f2d4b3192ae75fe7cd9092b, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:14.281272 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bf2906-d84a-41e3-9ac5-de7acc5af161 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:14.297430 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1438/2885] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:27:14 2022] PUT /identity/v3/projects/9392602c02724740bef91b490d074c3c/users/8ffbd6a35f2d4b3192ae75fe7cd9092b/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:14.309163 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9977b7c9-4ef0-485a-b70e-86abcc8d15bd None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:14.310000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9977b7c9-4ef0-485a-b70e-86abcc8d15bd None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.310000 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9977b7c9-4ef0-485a-b70e-86abcc8d15bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:14.344273 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-9977b7c9-4ef0-485a-b70e-86abcc8d15bd None None] MFA Rules not processed for user `8ffbd6a35f2d4b3192ae75fe7cd9092b`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:14.345638 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9977b7c9-4ef0-485a-b70e-86abcc8d15bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:14.400815 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:14.429428 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1448/2886] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:14 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:14.434927 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bf2906-d84a-41e3-9ac5-de7acc5af161 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:14.520809 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bf2906-d84a-41e3-9ac5-de7acc5af161 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:14.528035 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5505b0-23dd-4244-96a8-e64aa3f1f997 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:14.528813 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5505b0-23dd-4244-96a8-e64aa3f1f997 admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:14.529191 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5505b0-23dd-4244-96a8-e64aa3f1f997 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.529570 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5505b0-23dd-4244-96a8-e64aa3f1f997 admin admin] PATH_INFO: `/v3/projects` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:14.530404 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5505b0-23dd-4244-96a8-e64aa3f1f997 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:14.532811 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5505b0-23dd-4244-96a8-e64aa3f1f997 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:14.573243 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1439/2887] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:27:14 2022] POST /identity/v3/projects => generated 385 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:27:14.578963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9977b7c9-4ef0-485a-b70e-86abcc8d15bd None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:14.592923 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9977b7c9-4ef0-485a-b70e-86abcc8d15bd None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:14.662696 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab9b9f9-f86b-4cad-878b-aa3a36128844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:14.663601 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab9b9f9-f86b-4cad-878b-aa3a36128844 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:14.663951 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab9b9f9-f86b-4cad-878b-aa3a36128844 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.664266 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab9b9f9-f86b-4cad-878b-aa3a36128844 admin admin] PATH_INFO: `/v3/users` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:14.665013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab9b9f9-f86b-4cad-878b-aa3a36128844 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:14.667099 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab9b9f9-f86b-4cad-878b-aa3a36128844 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:14.693369 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1449/2888] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:27:14 2022] POST /identity/v3/users => generated 299 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 31 01:27:14.697876 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5505b0-23dd-4244-96a8-e64aa3f1f997 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:14.709618 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5505b0-23dd-4244-96a8-e64aa3f1f997 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:14.718364 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4c2c5a-d83a-4d3f-a4cd-faa4b6b1bd3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:14.719389 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4c2c5a-d83a-4d3f-a4cd-faa4b6b1bd3c admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:14.719886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4c2c5a-d83a-4d3f-a4cd-faa4b6b1bd3c admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.720344 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4c2c5a-d83a-4d3f-a4cd-faa4b6b1bd3c admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:14.721680 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4c2c5a-d83a-4d3f-a4cd-faa4b6b1bd3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:14.724772 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4c2c5a-d83a-4d3f-a4cd-faa4b6b1bd3c admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:14.727500 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:14.749599 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1440/2889] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:27:14 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:27:14.754449 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab9b9f9-f86b-4cad-878b-aa3a36128844 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:14.765793 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab9b9f9-f86b-4cad-878b-aa3a36128844 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:14.770783 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee942edd-7bbb-462c-a78f-a434906b6054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:14.771663 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee942edd-7bbb-462c-a78f-a434906b6054 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:14.771989 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee942edd-7bbb-462c-a78f-a434906b6054 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.772313 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee942edd-7bbb-462c-a78f-a434906b6054 admin admin] PATH_INFO: `/v3/projects/db976478621e4246bdc18cb138673310/users/f5428b8cefad4a0fad39e648fdc5cdfb/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:14.790013 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee942edd-7bbb-462c-a78f-a434906b6054 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db976478621e4246bdc18cb138673310, user_id=f5428b8cefad4a0fad39e648fdc5cdfb, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:14.792763 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee942edd-7bbb-462c-a78f-a434906b6054 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:14.809774 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1450/2890] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:27:14 2022] PUT /identity/v3/projects/db976478621e4246bdc18cb138673310/users/f5428b8cefad4a0fad39e648fdc5cdfb/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:27:14.814284 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4c2c5a-d83a-4d3f-a4cd-faa4b6b1bd3c admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:14.826545 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4c2c5a-d83a-4d3f-a4cd-faa4b6b1bd3c admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:14.895935 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fd2dbe-600c-4b73-aeff-48bf355ca3e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:14.896965 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fd2dbe-600c-4b73-aeff-48bf355ca3e5 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:14.897459 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fd2dbe-600c-4b73-aeff-48bf355ca3e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.897965 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fd2dbe-600c-4b73-aeff-48bf355ca3e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:14.899291 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fd2dbe-600c-4b73-aeff-48bf355ca3e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:14.902479 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fd2dbe-600c-4b73-aeff-48bf355ca3e5 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:14.922887 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1441/2891] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:27:14 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:27:14.928460 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee942edd-7bbb-462c-a78f-a434906b6054 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:14.939433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee942edd-7bbb-462c-a78f-a434906b6054 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:14.945457 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82540065-7b75-480c-92b1-8180b06dcdb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:14.946199 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82540065-7b75-480c-92b1-8180b06dcdb6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:14.946518 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82540065-7b75-480c-92b1-8180b06dcdb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.946834 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82540065-7b75-480c-92b1-8180b06dcdb6 admin admin] PATH_INFO: `/v3/projects/db976478621e4246bdc18cb138673310/users/f5428b8cefad4a0fad39e648fdc5cdfb/roles/8814964206e14dc3ae6ab69e8d874f51` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:14.949133 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82540065-7b75-480c-92b1-8180b06dcdb6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db976478621e4246bdc18cb138673310, user_id=f5428b8cefad4a0fad39e648fdc5cdfb, role_id=8814964206e14dc3ae6ab69e8d874f51)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:14.952093 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82540065-7b75-480c-92b1-8180b06dcdb6 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:14.964367 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1451/2892] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:27:14 2022] PUT /identity/v3/projects/db976478621e4246bdc18cb138673310/users/f5428b8cefad4a0fad39e648fdc5cdfb/roles/8814964206e14dc3ae6ab69e8d874f51 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:27:14.977064 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33e2efe-b54f-4017-9e18-965c4afd86ba None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:14.977342 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33e2efe-b54f-4017-9e18-965c4afd86ba None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:14.977614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33e2efe-b54f-4017-9e18-965c4afd86ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:15.009680 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-c33e2efe-b54f-4017-9e18-965c4afd86ba None None] MFA Rules not processed for user `f5428b8cefad4a0fad39e648fdc5cdfb`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:15.011007 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c33e2efe-b54f-4017-9e18-965c4afd86ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:15.096524 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1442/2893] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:14 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:15.109321 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82540065-7b75-480c-92b1-8180b06dcdb6 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:15.111261 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-82540065-7b75-480c-92b1-8180b06dcdb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:15.182385 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82540065-7b75-480c-92b1-8180b06dcdb6 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:15.186357 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1ca5b0-7053-4233-8b87-1488ada7ab0b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:15.186989 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ca5b0-7053-4233-8b87-1488ada7ab0b service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:15.187267 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ca5b0-7053-4233-8b87-1488ada7ab0b service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:15.187572 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ca5b0-7053-4233-8b87-1488ada7ab0b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:15.242487 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:15.254301 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1ca5b0-7053-4233-8b87-1488ada7ab0b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:15.255845 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1ca5b0-7053-4233-8b87-1488ada7ab0b service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:15.266203 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1452/2894] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:27:15 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:27:16.908921 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33e2efe-b54f-4017-9e18-965c4afd86ba None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:16.910497 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-c33e2efe-b54f-4017-9e18-965c4afd86ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:16.924380 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33e2efe-b54f-4017-9e18-965c4afd86ba None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:16.929448 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ed0e61-4389-46d5-b4bb-12aa6f7e6712 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:16.930234 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ed0e61-4389-46d5-b4bb-12aa6f7e6712 service cinder] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:16.930614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ed0e61-4389-46d5-b4bb-12aa6f7e6712 service cinder] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:16.931011 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ed0e61-4389-46d5-b4bb-12aa6f7e6712 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:16.998894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ed0e61-4389-46d5-b4bb-12aa6f7e6712 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:17.000410 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ed0e61-4389-46d5-b4bb-12aa6f7e6712 service cinder] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:17.011413 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1443/2895] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:27:16 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:27:17.679063 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1ca5b0-7053-4233-8b87-1488ada7ab0b service cinder] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:17.691286 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1ca5b0-7053-4233-8b87-1488ada7ab0b service cinder] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:17.695471 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef44d29-f6a9-42c2-a05e-7909f1b8fd45 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:17.696107 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef44d29-f6a9-42c2-a05e-7909f1b8fd45 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:17.696388 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef44d29-f6a9-42c2-a05e-7909f1b8fd45 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:17.696661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef44d29-f6a9-42c2-a05e-7909f1b8fd45 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:17.709705 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef44d29-f6a9-42c2-a05e-7909f1b8fd45 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:17.711155 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef44d29-f6a9-42c2-a05e-7909f1b8fd45 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:17.722258 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1453/2896] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:27:17 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:27:21.847886 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ed0e61-4389-46d5-b4bb-12aa6f7e6712 service cinder] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:21.879948 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=100 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:21.932829 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ed0e61-4389-46d5-b4bb-12aa6f7e6712 service cinder] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:21.937583 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:21.938516 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:21.938825 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] PATH_INFO: `/v3/users/def7b60ff1d440f1bfc74c43da96e928` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:21.940345 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=def7b60ff1d440f1bfc74c43da96e928)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:21.942543 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:22.020069 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'def7b60ff1d440f1bfc74c43da96e928'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.021116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'def7b60ff1d440f1bfc74c43da96e928'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.032372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'def7b60ff1d440f1bfc74c43da96e928'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.052681 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'def7b60ff1d440f1bfc74c43da96e928'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.060419 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'def7b60ff1d440f1bfc74c43da96e928'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.067953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1444/2897] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:27:21 2022] DELETE /identity/v3/users/def7b60ff1d440f1bfc74c43da96e928 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:22.072789 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146a1db8-7a6c-4882-8681-604dfbc155d7 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:22.073402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146a1db8-7a6c-4882-8681-604dfbc155d7 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:22.073402 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146a1db8-7a6c-4882-8681-604dfbc155d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:22.094268 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-146a1db8-7a6c-4882-8681-604dfbc155d7 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:22.095589 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-146a1db8-7a6c-4882-8681-604dfbc155d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:22.166053 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1454/2898] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:22.180025 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:22.182000 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:22.256293 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7111ae9a-e631-49a4-bd75-cbbe585272a9 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:22.260164 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9367e4b-59ea-4aec-a865-cac34b7e1f92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:22.260797 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9367e4b-59ea-4aec-a865-cac34b7e1f92 service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:22.261071 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9367e4b-59ea-4aec-a865-cac34b7e1f92 service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:22.261338 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9367e4b-59ea-4aec-a865-cac34b7e1f92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:22.274128 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9367e4b-59ea-4aec-a865-cac34b7e1f92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:22.275763 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9367e4b-59ea-4aec-a865-cac34b7e1f92 service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:22.287537 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1445/2899] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:27:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 31 01:27:22.625913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146a1db8-7a6c-4882-8681-604dfbc155d7 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:22.627782 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-146a1db8-7a6c-4882-8681-604dfbc155d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:22.648619 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146a1db8-7a6c-4882-8681-604dfbc155d7 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:22.666814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:22.667724 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:22.667894 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:22.668058 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] PATH_INFO: `/v3/projects/f0a560fca7d54959afb8cc66655624dc` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:22.669631 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0a560fca7d54959afb8cc66655624dc)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:22.671728 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:22.726673 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0a560fca7d54959afb8cc66655624dc', 'request_id': 'req-9897461f-2661-4d6a-adbb-516e75d4e5c9'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.733565 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0a560fca7d54959afb8cc66655624dc', 'request_id': 'req-9897461f-2661-4d6a-adbb-516e75d4e5c9'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.739982 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0a560fca7d54959afb8cc66655624dc', 'request_id': 'req-9897461f-2661-4d6a-adbb-516e75d4e5c9'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.755207 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] The token cache is being invalidate because project f0a560fca7d54959afb8cc66655624dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:27:22.755693 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.756571 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.758330 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1455/2900] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:27:22 2022] DELETE /identity/v3/projects/f0a560fca7d54959afb8cc66655624dc => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:22.762381 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9367e4b-59ea-4aec-a865-cac34b7e1f92 service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:22.773455 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9367e4b-59ea-4aec-a865-cac34b7e1f92 service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:22.814221 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:22.851923 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:22.853115 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:22.853539 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:22.854082 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] PATH_INFO: `/v3/users/8ffbd6a35f2d4b3192ae75fe7cd9092b` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:22.855870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ffbd6a35f2d4b3192ae75fe7cd9092b)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:22.858481 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:22.923431 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ffbd6a35f2d4b3192ae75fe7cd9092b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.924008 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ffbd6a35f2d4b3192ae75fe7cd9092b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.933193 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ffbd6a35f2d4b3192ae75fe7cd9092b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.948072 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ffbd6a35f2d4b3192ae75fe7cd9092b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.956290 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ffbd6a35f2d4b3192ae75fe7cd9092b'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:22.963010 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1446/2901] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:27:22 2022] DELETE /identity/v3/users/8ffbd6a35f2d4b3192ae75fe7cd9092b => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:23.369695 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:23.371611 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:23.392963 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9897461f-2661-4d6a-adbb-516e75d4e5c9 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:23.462014 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:23.462682 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:23.462967 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:23.463246 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] PATH_INFO: `/v3/projects/9392602c02724740bef91b490d074c3c` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:23.464499 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9392602c02724740bef91b490d074c3c)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:23.466466 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:23.469287 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:23.524969 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9392602c02724740bef91b490d074c3c', 'request_id': 'req-bc49d061-5358-4a13-88e1-cea7daec6107'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:23.532062 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9392602c02724740bef91b490d074c3c', 'request_id': 'req-bc49d061-5358-4a13-88e1-cea7daec6107'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:23.541083 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9392602c02724740bef91b490d074c3c', 'request_id': 'req-bc49d061-5358-4a13-88e1-cea7daec6107'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:23.559913 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] The token cache is being invalidate because project 9392602c02724740bef91b490d074c3c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:27:23.560206 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:23.561078 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:23.562671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1456/2902] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:27:23 2022] DELETE /identity/v3/projects/9392602c02724740bef91b490d074c3c => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:27:23.567762 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:23.569586 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:23.585360 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dee02e0-3576-44ad-8a02-c8e821388724 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:23.658817 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb734c9-e0fa-4992-b967-ebe64a4708bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:23.659532 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb734c9-e0fa-4992-b967-ebe64a4708bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:23.659822 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb734c9-e0fa-4992-b967-ebe64a4708bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:23.660099 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb734c9-e0fa-4992-b967-ebe64a4708bc admin admin] PATH_INFO: `/v3/users/f5428b8cefad4a0fad39e648fdc5cdfb` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:23.662155 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb734c9-e0fa-4992-b967-ebe64a4708bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=f5428b8cefad4a0fad39e648fdc5cdfb)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:23.664163 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb734c9-e0fa-4992-b967-ebe64a4708bc admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:23.721446 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-edb734c9-e0fa-4992-b967-ebe64a4708bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f5428b8cefad4a0fad39e648fdc5cdfb'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:23.722220 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-edb734c9-e0fa-4992-b967-ebe64a4708bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f5428b8cefad4a0fad39e648fdc5cdfb'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:23.731953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-edb734c9-e0fa-4992-b967-ebe64a4708bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f5428b8cefad4a0fad39e648fdc5cdfb'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:23.740107 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:23.754578 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-edb734c9-e0fa-4992-b967-ebe64a4708bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f5428b8cefad4a0fad39e648fdc5cdfb'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:23.763458 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-edb734c9-e0fa-4992-b967-ebe64a4708bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f5428b8cefad4a0fad39e648fdc5cdfb'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:23.771600 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1447/2903] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:27:23 2022] DELETE /identity/v3/users/f5428b8cefad4a0fad39e648fdc5cdfb => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:24.180857 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:24.182433 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:24.199819 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc49d061-5358-4a13-88e1-cea7daec6107 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:24.284066 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a26fbd4-af82-446c-ba18-908eb422f624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:24.285068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a26fbd4-af82-446c-ba18-908eb422f624 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:24.285559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a26fbd4-af82-446c-ba18-908eb422f624 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:24.286049 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a26fbd4-af82-446c-ba18-908eb422f624 admin admin] PATH_INFO: `/v3/projects/db976478621e4246bdc18cb138673310` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:24.288219 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a26fbd4-af82-446c-ba18-908eb422f624 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db976478621e4246bdc18cb138673310)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:24.292363 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a26fbd4-af82-446c-ba18-908eb422f624 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:24.362736 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-6a26fbd4-af82-446c-ba18-908eb422f624 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db976478621e4246bdc18cb138673310', 'request_id': 'req-6a26fbd4-af82-446c-ba18-908eb422f624'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:24.370127 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-6a26fbd4-af82-446c-ba18-908eb422f624 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db976478621e4246bdc18cb138673310', 'request_id': 'req-6a26fbd4-af82-446c-ba18-908eb422f624'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:24.376835 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-6a26fbd4-af82-446c-ba18-908eb422f624 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db976478621e4246bdc18cb138673310', 'request_id': 'req-6a26fbd4-af82-446c-ba18-908eb422f624'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:24.393339 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-6a26fbd4-af82-446c-ba18-908eb422f624 admin admin] The token cache is being invalidate because project db976478621e4246bdc18cb138673310 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:27:24.393846 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-6a26fbd4-af82-446c-ba18-908eb422f624 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:24.394979 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-6a26fbd4-af82-446c-ba18-908eb422f624 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:24.398772 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1457/2904] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:27:24 2022] DELETE /identity/v3/projects/db976478621e4246bdc18cb138673310 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:24.406945 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94c18c7-9fa7-4faa-a87c-982fb4338851 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:24.407607 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94c18c7-9fa7-4faa-a87c-982fb4338851 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:24.407962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94c18c7-9fa7-4faa-a87c-982fb4338851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:24.436042 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-d94c18c7-9fa7-4faa-a87c-982fb4338851 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:24.437622 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-d94c18c7-9fa7-4faa-a87c-982fb4338851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:24.517484 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1448/2905] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:24.527068 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e431c9d-cb7a-4e8f-bd07-e66cad001078 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:24.527519 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e431c9d-cb7a-4e8f-bd07-e66cad001078 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:24.527956 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e431c9d-cb7a-4e8f-bd07-e66cad001078 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:24.562770 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-5e431c9d-cb7a-4e8f-bd07-e66cad001078 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:24.565271 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-5e431c9d-cb7a-4e8f-bd07-e66cad001078 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:24.580618 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1458/2906] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:24.585476 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94c18c7-9fa7-4faa-a87c-982fb4338851 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:24.595777 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94c18c7-9fa7-4faa-a87c-982fb4338851 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:24.600691 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec1fa06-c869-4d73-a6ee-ff4b2c25ecea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:24.601340 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec1fa06-c869-4d73-a6ee-ff4b2c25ecea admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:24.601619 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec1fa06-c869-4d73-a6ee-ff4b2c25ecea admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:24.601894 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec1fa06-c869-4d73-a6ee-ff4b2c25ecea admin admin] PATH_INFO: `/v3/domains` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:24.602608 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec1fa06-c869-4d73-a6ee-ff4b2c25ecea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80209) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 31 01:27:24.602918 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec1fa06-c869-4d73-a6ee-ff4b2c25ecea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:24.604614 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec1fa06-c869-4d73-a6ee-ff4b2c25ecea admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:24.619453 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1449/2907] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Jul 31 01:27:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 31 01:27:24.624244 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e431c9d-cb7a-4e8f-bd07-e66cad001078 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:24.635803 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e431c9d-cb7a-4e8f-bd07-e66cad001078 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:24.641651 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eed612-5bdb-4e57-b761-987e707bd379 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:24.642442 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eed612-5bdb-4e57-b761-987e707bd379 admin admin] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:24.643001 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eed612-5bdb-4e57-b761-987e707bd379 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:24.643444 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eed612-5bdb-4e57-b761-987e707bd379 admin admin] PATH_INFO: `/v3/projects` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:24.644275 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59eed612-5bdb-4e57-b761-987e707bd379 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:24.646653 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59eed612-5bdb-4e57-b761-987e707bd379 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:24.663928 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:24.685262 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1459/2908] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Jul 31 01:27:24 2022] POST /identity/v3/projects => generated 373 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:27:24.687086 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec1fa06-c869-4d73-a6ee-ff4b2c25ecea admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:24.698970 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec1fa06-c869-4d73-a6ee-ff4b2c25ecea admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:24.766870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fab1756-8828-4783-9037-b44df86db1bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:24.767705 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fab1756-8828-4783-9037-b44df86db1bf admin admin] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:24.767989 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fab1756-8828-4783-9037-b44df86db1bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:24.768266 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fab1756-8828-4783-9037-b44df86db1bf admin admin] PATH_INFO: `/v3/users` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:24.768967 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fab1756-8828-4783-9037-b44df86db1bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:24.770955 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fab1756-8828-4783-9037-b44df86db1bf admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:24.795028 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1450/2909] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Jul 31 01:27:24 2022] POST /identity/v3/users => generated 287 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 31 01:27:24.799633 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eed612-5bdb-4e57-b761-987e707bd379 admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:24.812361 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eed612-5bdb-4e57-b761-987e707bd379 admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:24.817542 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a88c8cb-67d9-4f1d-85bc-e83f1f7bacec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:24.818178 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a88c8cb-67d9-4f1d-85bc-e83f1f7bacec admin admin] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:24.818562 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a88c8cb-67d9-4f1d-85bc-e83f1f7bacec admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:24.818836 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a88c8cb-67d9-4f1d-85bc-e83f1f7bacec admin admin] PATH_INFO: `/v3/roles` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:24.819613 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a88c8cb-67d9-4f1d-85bc-e83f1f7bacec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:24.821332 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a88c8cb-67d9-4f1d-85bc-e83f1f7bacec admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:24.832670 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1460/2910] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:27:24 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:27:24.838627 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fab1756-8828-4783-9037-b44df86db1bf admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:24.850863 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fab1756-8828-4783-9037-b44df86db1bf admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:24.856184 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab9ae87-0b28-4d4e-8b86-daa88c9d09e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:24.856870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab9ae87-0b28-4d4e-8b86-daa88c9d09e7 admin admin] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:24.857216 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab9ae87-0b28-4d4e-8b86-daa88c9d09e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:24.857528 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab9ae87-0b28-4d4e-8b86-daa88c9d09e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:24.858279 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab9ae87-0b28-4d4e-8b86-daa88c9d09e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:24.860116 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab9ae87-0b28-4d4e-8b86-daa88c9d09e7 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:24.874912 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1451/2911] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Jul 31 01:27:24 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 31 01:27:24.881042 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a88c8cb-67d9-4f1d-85bc-e83f1f7bacec admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:24.894661 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a88c8cb-67d9-4f1d-85bc-e83f1f7bacec admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:24.900120 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8318ce48-427d-4565-bbb9-863d15fdc32f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:24.901081 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8318ce48-427d-4565-bbb9-863d15fdc32f admin admin] REQUEST_METHOD: `PUT` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:24.901456 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8318ce48-427d-4565-bbb9-863d15fdc32f admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:24.901838 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8318ce48-427d-4565-bbb9-863d15fdc32f admin admin] PATH_INFO: `/v3/projects/8af1ddbb4c2c40afa89a52b3d354f3e4/users/9ca984a8a9a849d69ef7b6850d0ff0bd/roles/ac68e1820c604122981fd5c4e2ebc447` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:24.922221 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8318ce48-427d-4565-bbb9-863d15fdc32f admin admin] RBAC: Authorizing `identity:create_grant(project_id=8af1ddbb4c2c40afa89a52b3d354f3e4, user_id=9ca984a8a9a849d69ef7b6850d0ff0bd, role_id=ac68e1820c604122981fd5c4e2ebc447)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:24.925300 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8318ce48-427d-4565-bbb9-863d15fdc32f admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:24.941582 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1461/2912] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Jul 31 01:27:24 2022] PUT /identity/v3/projects/8af1ddbb4c2c40afa89a52b3d354f3e4/users/9ca984a8a9a849d69ef7b6850d0ff0bd/roles/ac68e1820c604122981fd5c4e2ebc447 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:24.952609 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6827640-dac6-490b-b450-2676aae6e2f4 None None] REQUEST_METHOD: `POST` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:24.953044 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6827640-dac6-490b-b450-2676aae6e2f4 None None] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:24.953426 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6827640-dac6-490b-b450-2676aae6e2f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:24.991719 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.auth.core [None req-b6827640-dac6-490b-b450-2676aae6e2f4 None None] MFA Rules not processed for user `9ca984a8a9a849d69ef7b6850d0ff0bd`. Rule list: `[]` (Enabled: `True`). {{(pid=80209) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:24.992962 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-b6827640-dac6-490b-b450-2676aae6e2f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:25.028814 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:25.072397 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1452/2913] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:24 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:25.083907 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8318ce48-427d-4565-bbb9-863d15fdc32f admin admin] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:25.085715 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-8318ce48-427d-4565-bbb9-863d15fdc32f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:25.154552 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8318ce48-427d-4565-bbb9-863d15fdc32f admin admin] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:25.161307 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ee0f8d-4357-44ff-8fed-1602a483a688 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c8338bf95d34b228176b556afd93155', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:25.162377 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ee0f8d-4357-44ff-8fed-1602a483a688 service cinder] REQUEST_METHOD: `GET` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:25.162871 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ee0f8d-4357-44ff-8fed-1602a483a688 service cinder] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:25.163326 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ee0f8d-4357-44ff-8fed-1602a483a688 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:25.183707 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ee0f8d-4357-44ff-8fed-1602a483a688 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:25.185886 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ee0f8d-4357-44ff-8fed-1602a483a688 service cinder] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:25.197568 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1462/2914] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:27:25 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:27:35.069122 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=13 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:35.192765 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:INSERT=2,keystone:SELECT=43 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:36.250870 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6827640-dac6-490b-b450-2676aae6e2f4 None None] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:36.325257 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6827640-dac6-490b-b450-2676aae6e2f4 None None] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:36.330379 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:36.331075 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:36.331370 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:36.331657 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] PATH_INFO: `/v3/users/9ca984a8a9a849d69ef7b6850d0ff0bd` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:36.333486 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9ca984a8a9a849d69ef7b6850d0ff0bd)` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:36.335512 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:36.392511 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9ca984a8a9a849d69ef7b6850d0ff0bd'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:36.393214 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9ca984a8a9a849d69ef7b6850d0ff0bd'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:36.402118 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9ca984a8a9a849d69ef7b6850d0ff0bd'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:36.417577 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9ca984a8a9a849d69ef7b6850d0ff0bd'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:36.426953 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.notifications [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9ca984a8a9a849d69ef7b6850d0ff0bd'} {{(pid=80209) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:36.435215 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1453/2915] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Jul 31 01:27:36 2022] DELETE /identity/v3/users/9ca984a8a9a849d69ef7b6850d0ff0bd => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 31 01:27:36.441890 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b850ff-e849-4dd7-8316-d6c8680a2200 None None] REQUEST_METHOD: `POST` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:36.442656 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b850ff-e849-4dd7-8316-d6c8680a2200 None None] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:36.442656 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b850ff-e849-4dd7-8316-d6c8680a2200 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:36.464432 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.auth.core [None req-a3b850ff-e849-4dd7-8316-d6c8680a2200 None None] MFA Rules not processed for user `e711c7fc76434a0b86b92dc8bcf69e87`. Rule list: `[]` (Enabled: `True`). {{(pid=80208) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 31 01:27:36.465671 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a3b850ff-e849-4dd7-8316-d6c8680a2200 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:36.543559 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1463/2916] 10.222.0.42 () {58 vars in 958 bytes} [Sun Jul 31 01:27:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 31 01:27:36.557149 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:36.559392 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:36.665058 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8141b1-5ca6-46b8-9594-9e520053f616 admin admin] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:36.669288 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcc99e3-b30d-487a-ba8d-242888d0eb5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94e868131d33472b9ded8486274a07b9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:36.669980 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcc99e3-b30d-487a-ba8d-242888d0eb5a service neutron] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:36.670303 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcc99e3-b30d-487a-ba8d-242888d0eb5a service neutron] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:36.670619 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcc99e3-b30d-487a-ba8d-242888d0eb5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:36.681343 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcc99e3-b30d-487a-ba8d-242888d0eb5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:36.682863 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcc99e3-b30d-487a-ba8d-242888d0eb5a service neutron] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:36.695432 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1454/2917] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:27:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:27:37.078924 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b850ff-e849-4dd7-8316-d6c8680a2200 None None] Authenticating user token {{(pid=80208) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:27:37.081434 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.fernet_utils [None req-a3b850ff-e849-4dd7-8316-d6c8680a2200 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80208) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:27:37.104019 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b850ff-e849-4dd7-8316-d6c8680a2200 None None] Validating token access rules against request {{(pid=80208) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:27:37.109841 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be5705a-2728-46a1-90bb-2febadbaafc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e711c7fc76434a0b86b92dc8bcf69e87', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '18e627c596bb4c99b66fda211d0e4785', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80208) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:27:37.110514 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be5705a-2728-46a1-90bb-2febadbaafc5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:27:37.110792 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be5705a-2728-46a1-90bb-2febadbaafc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:27:37.111066 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be5705a-2728-46a1-90bb-2febadbaafc5 admin admin] PATH_INFO: `/v3/projects/8af1ddbb4c2c40afa89a52b3d354f3e4` {{(pid=80208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:27:37.112327 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be5705a-2728-46a1-90bb-2febadbaafc5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8af1ddbb4c2c40afa89a52b3d354f3e4)` {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:27:37.114854 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be5705a-2728-46a1-90bb-2febadbaafc5 admin admin] RBAC: Authorization granted {{(pid=80208) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:27:37.176932 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8be5705a-2728-46a1-90bb-2febadbaafc5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8af1ddbb4c2c40afa89a52b3d354f3e4', 'request_id': 'req-8be5705a-2728-46a1-90bb-2febadbaafc5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:37.184814 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8be5705a-2728-46a1-90bb-2febadbaafc5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8af1ddbb4c2c40afa89a52b3d354f3e4', 'request_id': 'req-8be5705a-2728-46a1-90bb-2febadbaafc5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:37.192794 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8be5705a-2728-46a1-90bb-2febadbaafc5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8af1ddbb4c2c40afa89a52b3d354f3e4', 'request_id': 'req-8be5705a-2728-46a1-90bb-2febadbaafc5'} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:37.207784 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8be5705a-2728-46a1-90bb-2febadbaafc5 admin admin] The token cache is being invalidate because project 8af1ddbb4c2c40afa89a52b3d354f3e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80208) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 31 01:27:37.208113 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8be5705a-2728-46a1-90bb-2febadbaafc5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:37.208877 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG keystone.notifications [None req-8be5705a-2728-46a1-90bb-2febadbaafc5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80208) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 31 01:27:37.210465 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: [pid: 80208|app: 0|req: 1464/2918] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Jul 31 01:27:37 2022] DELETE /identity/v3/projects/8af1ddbb4c2c40afa89a52b3d354f3e4 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 31 01:27:46.689615 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:27:47.204171 dv-cinder-master-3-iscsi devstack@keystone.service[80208]: DEBUG dbcounter [-] [80208] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80208) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 31 01:32:11.354083 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcc99e3-b30d-487a-ba8d-242888d0eb5a service neutron] Authenticating user token {{(pid=80209) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 31 01:32:11.356289 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-7fcc99e3-b30d-487a-ba8d-242888d0eb5a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:32:11.459125 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcc99e3-b30d-487a-ba8d-242888d0eb5a service neutron] Validating token access rules against request {{(pid=80209) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 31 01:32:11.529729 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1843d00-1232-4e24-835b-7c6844b262f7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1d562ccd50e4492281a9f435f809d49b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8b41573c9f3146d19a306089d686015c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80209) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 31 01:32:11.530372 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1843d00-1232-4e24-835b-7c6844b262f7 service placement] REQUEST_METHOD: `GET` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 31 01:32:11.530645 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1843d00-1232-4e24-835b-7c6844b262f7 service placement] SCRIPT_NAME: `/identity` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 31 01:32:11.530917 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1843d00-1232-4e24-835b-7c6844b262f7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80209) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 31 01:32:11.532963 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.fernet_utils [None req-e1843d00-1232-4e24-835b-7c6844b262f7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80209) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 31 01:32:11.547158 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1843d00-1232-4e24-835b-7c6844b262f7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 31 01:32:11.549735 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1843d00-1232-4e24-835b-7c6844b262f7 service placement] RBAC: Authorization granted {{(pid=80209) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 31 01:32:11.566799 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: [pid: 80209|app: 0|req: 1455/2919] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Jul 31 01:32:11 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 31 01:32:21.557926 dv-cinder-master-3-iscsi devstack@keystone.service[80209]: DEBUG dbcounter [-] [80209] Writing DB stats keystone:SELECT=58 {{(pid=80209) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}